Rather than writing another review today, this Tuesday’s essay is an exploration of what I’m looking for when reviewing a report.
I started the ‘Too Long: Did Read’ series of blog post at the suggestion of a friend and former coworker. I have written, edited, and led the charge of over 30 industry reports, the majority with her help. She’s heard me critique the work of other organizations and be at least as critical of our own works many times over the years. “Why not use that experience as fuel for blog posts?”, she asked.
I’ve been following Verizon’s Data Breach Investigation Report (DBIR) since the first volume was published. I had the privilege of contributing to the report while working at Verizon and contributed data to the DBIR as part of my role at Akamai. I helped create the first version of Akamai’s State of the Internet / Security (SOTI) and led the effort through the beginning of 2022. What I’m getting at is that I’m not some rando talking smack about reports in the security industry. I have experience to draw from directly related to these reviews.
What am I looking for?
When I read any of the reports, there’s a laundry list of things I’m looking for. More than any other factor I’m looking for a report that understands it’s audience and gives a compelling reason to read past the introduction. This may sound easy and obvious, but it is probably the single biggest issue most reports in our industry fail to do. I’ve failed to provide this in many of my own early efforts. The reader is being asked to spend one of their most precious commodities, time, so they need know from the start it will be time well spent.
Whether we’re talking about sales or writing a book, this is called the ‘hook’. No big surprise there, any good marketing team or PR will ask the author for this hook. It gives them a better idea of how to use the content, what press outlets or reporters to approach, and makes their own work more effective. In my experience, the hook should be the very first thing the reader sees! More important than the index, an introduction to the author, or any other part of the report, I want to see something that tells me why I should turn to page 2! A good understanding of what the author is trying to educate the reader about is also going to be one of the biggest things that will get a reader to open the PDF of your report in the first place.
I learned to ask myself one question after the writing was done, as the report went to design: What three things do I want my readers to walk away with if they quit reading after the first page of the report? These can be part of the introduction, as a standalone section next to the intro, or in some other form ON THE FIRST PAGE. There might be two bullet points or five, the count isn’t important. After you’ve written ten, twenty, or more pages, it’s easy to lose the core of your message in the fog of relief from getting the first draft completed.
What’s inside?
Once I know what the purpose of report is, how well it communicates this purpose is next on the block. If I have read the first half dozen pages and still haven’t been able to divine its purpose, it’s a major failure. In the land of report writing, there’s no room for subtlety. Tell me what you’re going to tell me, tell it to me, then summarize it again at the end. Don’t hide your intelligence, make it as clear and straight forward as possible.
As I read through the pages, I need to know that the contents in the tin deliver on what the label advertised! If you’re selling your report as a technical treatise on DDoS attacks, I want to see in-depth information about attacks as quickly as possible. Sticking with a set format that starts the DDoS section on page 20 of a 50-page report is a recipe for failure. Not that I’ve ever done exactly that, of course. Even worse is when a report is really about a different topic than promised in the title, with only a nod to the title and introduction buried deep in the report.
How does it look?
Next, I look at the data visualizations: the plots, the charts, the diagrams used to reinforce the analysis in the report. Data visualization is a whole field on its own, and I can only brush against the surface of what there is to learn. I want to see a chart that directly relate to the analysis on the page. I want charts that are readable by the widest audience possible. I want charts that tell stories beyond what’s in the analysis.
I absolutely hate plots that are simply window dressing and have nothing to do with the content and analysis! Space is at a premium, time is valuable, so why waste both with something that offers nothing to the reader? If the author isn’t presenting analysis or providing additional meaning from an image in the report, why is it there? Having created a beautiful visualization isn’t a good enough reason to include the image if it doesn’t further the story. Personally, I want graphics to have descriptions and figure numbers as much as possible. I like to include additional analysis in the description, but that might not work for all authors.
I’m a huge advocate for using a Color Blind Friendly (CBF) palette. I have family, friends, and former co-workers who have various degrees and type of color blindness, and the thought of producing a report that doesn’t take that into account is anathema to me. Approximately 5% of the population is color blind, and more people suffer from color deficiency, a very similar problem. Color is also an incredibly valuable method of conveying information, so be sure to make the most of it. Coloring for Colorblindness by David Nichols is a good starting point for more information on making accessible plots.
Similarly, unless your audience is other data scientists, I prefer simple plots wherever possible. River plots, dot plots, violin charts, matrix plots and all the other complex graphics are appealing to other data viz geeks, but indecipherable to the average reader. Most readers will skip a plot they have to struggle with rather than learn something from it. I once spent a page and a half of a 30-page report explaining how to read a river plot many years ago and learned this lesson the hard way.
Stick to bar charts, line plots, and other visualizations most readers can understand at a glance. I’ll even include pie charts in this suggestion, but only if the data has four or fewer data points that are highly dissimilar. This means you can’t use a pie chart to show data with 20 different points, each less than 5% of the total. A table might not be pretty but is much more accessible and educational for a reader.
It’s okay to use more complex plots, but only with forethought and a clear understanding of the point you’re trying to make. I strongly recommend reading ‘The Truthful Art’ by Alberto Cairo or ‘Storytelling with Data’ by Cole Nussbaumer Knaflic as good starting points. Data-Driven Security by Jacobs & Rudis is great for domain specific visualization. Yes, Mr. Tufte was once considered groundbreaking for his visualizations, but there are plenty of authors more in tune with current technology and methods of communicating data visually.
The Bottom Line
I could write a book on the topic of industry reports. I could, but it would be crap. Do expect more blog posts on the topic in the future; there’s still much more I could wax poetic about. Instead, I’d like to leave you with three questions an author should be asking as they begin the process of writing:
- Who am I writing for? An executive rarely has time to read more than the introduction, while the red team wants all the juicy details of the latest attack type. Write appropriately for your audience.
- What do I want them to walk thinking about? I cannot stress enough that if you haven’t communicated your primary message in the first 500 words, you’ve probably lost half your audience. If you can’t tell a reporter your why your publication is important to his audience, the chance of coverage for your story drops dramatically.
- Have I communicated my findings well? Whether it’s the analysis or the graphics in a report, if they leave the reader confused, you’ve failed at your most important task.
One last thing: A good editor is your best friend when it comes to writing! And like any good friend, they may sometimes tell you your baby is ugly. At least you can throw your first draft in the garbage and start over, unlike a baby.