It’s a great effort and the team that composed this clearly spent hours trying to get it right. It’s unfortunate that the old adage of “garbage in, garbage out” applies so strongly here. The data this is based on is either sketchy or completely missing. This is not the fault of the team but rather a symptom of the subject area. The report is well written, the graphs are informative, but there’s nothing that can save it from the fact that it’s based on at most 17% of ransomware attacks in Europe, UK and USA. There are no conclusions that can be drawn from such a small sample size. Kudos to the team, they don’t actually try to draw any, except that as a society we need to be better at reporting and documenting these attacks, otherwise we are never going to improve.
There’s a lot of secrecy around ransomeware. I am sure there’s a million reasons for it, a few come to mind immediately:
1. Shame - you got attacked, therefore you did something wrong. I can see us trying to work that one through as a society, since it is in essence victim blaming.
2. Fear - you got attacked, therefore you are vulnerable, therefore it will happen again. I think this one is surmountable as well, it's a culture shift, don't get me wrong, but the more we learn from the attacks, the stronger we become.
3. Shareholders - I am by no means an expert in investment, shares and boards, but I am going to assume that being attacked isn't rocket fuel for share prices. This one is a challenge, since markets work in mysterious ways and there's no way I can say that reporting an attack would be beneficial. I'd like it to be.
4. Confusion - you got attacked, what are you supposed to do exactly? Is this a 999 thing? Is there a different number? This one is probably the easiest one to tackle. Even if I am the first to admit that corporate training videos put me to sleep.
Mostly this report made me sad. A lot of effort, a lot of goodwill and the main thing I got out of it was that we don’t know much. And that paying the ransom probably works. Sad 😞
P.S. Also interesting that in one case decryption took such a long time, that even though the victim paid the ransom, there was still a lot of disruption.
]]>With that out of the way, this is the story of how adding a ‘Not found’ page in one of our Express.js applications took hours, and involved me learning way too much about routing in both Express and React, none of which actually helped.
We have an app. It started off as a React app with client-side routing. At one point we realised React was the wrong technology for our users (if you’re interested, I go in more detail here), so we started moving off it. There was no appetite to rewrite the React app as server side in one go, we had pretty much just released the final tweaks which broke the camel’s back. It did not make sense to continue building in React, so we switched to server-side routing and rendering. When I say switched, I don’t mean we switched, I mean we started using both.
Our express setup had something like this in it:
app.use(routes())
app.get('*', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
routes
is a file that has all the server-side routes in it. So if a route does not match any of the server-side ones, we call the react application. That’s worked great so far and we haven’t had a problem
Except now we wanted a ‘Not found’ page. The internet suggests we should do something like the following in Express:
// Handle 404
app.use(function(req, res) {
res.send('404: Page not Found', 404);
});
This would work great if we didn’t already have a catch-all path to serve our React app. I tried the above and (as is obvious to me now) completely shut off the React side of the site. I don’t love that code, but I don’t dislike it that much.
You might be thinking, well, that’s fine, don’t define the 404 in Express, leave the server-side routing as is and let React handle it. Credit goes to my colleague Jamie for suggesting this to me.
I think this is a brilliant idea, and it will probably work for you. If it does, you should definitely do that. Define a catch-all route at the end of your React Router setup and let that handle the ‘Not found’ behaviour.
That’s apparently not how we roll. This is a shortened version of the last route in our React Router setup:
<Route
render={() => (
<div className="content">
<ScrollToTop>
<Notifications />
<Route
render={({ location }) => {
if (config && config.googleAnalyticsId) {
ReactGA.pageview(location.pathname)
}
return (
<Header
/>
)
}}
/>
<FooterContainer feedbackEmail={config.mailTo} />
</ScrollToTop>
</div>
)}
/>
We already have a catch-all route. And it servers the layout, while all other routes populate the content within.
The only way we can do what we want is by letting one of the routers know about all of the paths, even ones it isn’t serving. This is not pretty, and it is not good, and it’s making the smell of technical debt stronger. And it works:
app.get('/react-route-1*', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
app.get('/react-route-2*', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
app.get('/react-route-3*', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
app.use('/react-route-4*', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
app.get('/react-route-5*', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
app.get('/react-route-6', (req, res) => {
res.sendFile(path.join(__dirname, '../build/index.html'))
})
// This is the 404 logic
app.use((req, res) => {
// Handle 404 as needed
})
I am definitely not proud of this. I hope you haven’t found this blog because you have the same problem. I really hope so.
]]>The final config looked like this:
# package.json
"devDependencies": {
...
"process": "0.11.10"
}
# webpack.config.js
module.exports = {
...
plugins: [
new webpack.ProvidePlugin({
process: 'process/browser',
}),
],
}
Let me preface this by saying I am not a Webpack/Javascript guru. Far from it. I have very little prior knowledge and rely solely on documentation and web search skills to get me to the goal.
Due to a recent security vulnerability in one of our webpack plugins, we decided to look into upgrading from Webpack 4.43.x to Webpack 5. We fully expected there to be issues, it’s a major version after all. However, the rabbit hole I fell into was slightly different than what I had envisioned.
First of all, what is Webpack. If you develop in the Javascript world, you probably know about it, if you don’t, it’s an asset bundler. It takes all of your static assets and, well, bundles them for better delivery and performance. It is a very popular, well-loved and well-maintained open source project.
Because it’s such a nice project, it has a migration guide. I was thrilled to see this. I was even more thrilled when I easily started checking things off it:
Amazing! All that’s left is to upgrade! So I did. And then the build didn’t run. Turns out, at the time of the upgrade, Webpack had just released a new beta version, which introduced a breaking change that affected one of our plugins. After a bit of digging, I pinned the webpack version to the one before that and reran the install and build. And it worked!
Except, there was a warning about path and polyfills. The upgrade guide does not mention anything about this, but Webpack 5 no longer auto loads polyfills for code node libraries. (Beta feedback). They did a community consultation on it and everything. The module that was erroring out was path
. The error suggested a course of action (aliasing to false), which worked, and the build succeeded.
The integration tests did not. Turns out there was this one page that used this one library, which was a third-level dependency that we couldn’t get rid of, which required the process
node library in the browser. This is where I spent almost 2 days. The library had an issue on it which at the time did not help me much. (I have since posted a comment with my findings to that issue, hoping to save at least one soul.) “Just polyfill process” meant nothing to me and it looked like that issue predated Webpack 5 anyway, so it wouldn’t have been a problem for the majority of people using the library.
I went on a wild goose chase trying to figure out how to polyfill process. My first problem was figuring out what the library I need was called. There are a few libraries named as if they’d be the right thing, but none of them worked. In the end I went and dug through Webpack code to find the list of modules they were no longer loading so I could find what the process one was. It’s process/browser
and the library is process. Cool, that’s fine. But how do you polyfill it?
I spent way too long moving things around in my our webpack config. Web searchers were not fruitful, it almost looked like nobody had needed to do this again. Granted, I may have been in a niche group, upgrading to a beta version and having a dependency I couldn’t get rid of which doesn’t work with that version without some kind of magical custom config.
To cut a very long and boring story short, try shimming instead. With the right library and the right config, I had the site working within 5 minutes.
]]>I am making the assumption that most of you have not been to a prison. I don’t mean as a prisoner, but also as a visitor, delivery person or in any kind of professional capacity. Statistically, I am probably correct.
I would like you to imagine a prison. Got it? Now, I would like you to imagine a Victorian prison. Because there’s plenty of those still in active service in the UK. If you are anything like me, the word “prison” fills you with dread and the concept of a Victorian prison is downright terrifying. Hold onto these feelings, they are very important.
What sorts of people may we find in a prison? Prisoners, probably. Yes, definitely prisoners. In fact, in some prisons, on some days, the ratio of prisoners to prison officers is 20:1. But there are other people in there, prison officers, medical staff. Workshop staff.
It’s about these non-prisoner prison dwellers that I’d like you to think for a second. Remember the feelings from a couple of minutes ago? Dread, terror? How do these feelings change when you think about the people who work in prisons? The people whose livelihood is a prison?
I’ve enumerated a few roles in prisons, but I’d like to expand on that so you can better understand who our users are. Prison staff have two main responsibilities, broadly - safety and rehabilitation. Safety covers a few aspects. Their personal safety, that of their colleagues. The safety of the offenders - from each other and also from themselves. Being an offender is not easy. And finally, the safety of the general public, ensuring procedures are followed and offenders are correctly monitored to avoid risk to the wider society. Safety plays a big part in what prison staff need to worry about.
Rehabilitation is the other side of the coin. The main goal of the Prison service is to reduce reoffending. It makes sense if you think about - if time spent in prison did not stop people offending, then is it really doing anything? The biggest factor in reducing reoffending is contact with people. Family visits are crucial. Phone calls are crucial. And time spent talking to prison staff is crucial. Prison staff can find out how an offender is doing from chatting to them, potentially avoiding a problem later on. If they have the trust of the offenders, they can find out gossip about who’s doing what where. And they can chat about the football on Saturday and how the new manager of team X sucks. Offenders are people, and people need social interaction.
There are around 135 establishments in the prison estate. Most policies are estate-wide, but they also leave room for maneuver and interpretation. Each prison’s no. 1 governor (that’s honestly what they’re called) is the ruler of their own castles. As a result every prison does something slightly different. Some prisons do a lot of things very differently. We’re building systems to rule them all. Except when we’re not. If only one prison has a certain issue, we’re probably not going to prioritise it. They’re probably not doing things entirely right. We’ll talk to them and try to retrain them. But possibly, just possibly, there’s a legitimate reason they’re doing something different. Maybe the layout of the prison is unique. Or one of the floors is no longer inhabited. Sometimes, we need to consider these things carefully. While we work with prisons as user units, they actually represent some 100s staff and potentially 1000s offenders. These are not small numbers when you think about the implications of not thoroughly evaluating the risks.
Enabling prison staff to spend as little time as possible in front of computers is our goal. The less time using our systems, the more time they can spend interacting with offenders, the better prisons are at rehabilitating. The title of this talk focuses on software and I’ve been talking a lot about feelings. That is because the main thing you need when building software for prison staff is empathy.
As part of my job I get to visit prisons from time to time. I’ve been to three so far - HMP Moorland, HMP Humber and HMP Leeds. Above everything else, above the jarring feeling of enclosement, the constant locking and unlocking of doors, the shouting, the level of alertness, above all of that, I left with one feeling - the determination to build the best damn software I can for these people because their work is already so incomprehensibly difficult. This determination is shared by everyone I work with. It’s quite a unique thing.
Shall we talk software? Let’s. We build web apps, so let’s talk browsers. Our browser support spans anything between IE8 and latest Chrome/Safari/Firefox. There is some fluctuation around how well some things work, but they work. Our users access our sites through a remote server (this is important later on) and as a rule have poor Internet connection. Page loads of any description are painful. There is no wifi in prisons, except for a couple of pilot sites and we have no mobile usage as mobile devices are still in pilot stages. In summary, desktop machines, every browser, remote server and slow internet. I have never catered for an environment less suited to the modern Web. But that’s just me.
Our tech stack is rather boring so I won’t dwell on it too much. Briefly: there’s an Oracle database somewhere in the nether, there are APIs over it, mostly in Java and Kotlin, and there are mostly stateless applications in Node over those APIs. This does not apply to everything, by a mile, but it’s common. We deploy to a kubernetes cluster using helm. As a full stack engineer I end up touching all of the above, for better or for worse. Have I used all the buzzwords? Did I say microservices? No? Well, them. And React. Our users don’t care about any of this, so we’re going to move on.
We try to avoid technical debt. And we try to avoid spaghetti. These things become a bit more difficult when we can’t necessarily afford to avoid edge cases. We don’t always win.
That’s the broad context. Let’s focus on a concrete example. The picture below is a screenshot of a page from one of our applications. This is the Whereabouts application, aka whereabouts are the offenders at any given time. The page title is the name of the activity location we’re looking at, in this case “Workshop 16 - Joinery”. This is data from our test environment, there is no real offender data on this screenshot, before you go and worry I’m exposing someone’s personal information.
Below the page heading, there is a date picker and a dropdown menu to select period - AM, PM or evening. These are horizontally aligned.
Below them is a link that says “Reload page”. Below that is a green button with text “Print list”. Underneath that there is a dropdown with the label “Order the list”. To the right of the dropdown are 4 pieces of text. The top two represent summary information: number of unique prisoners listed and number of sessions. Below those two are two links - attend all prisoners and all prisoners not required.
Underneath all of this, and really, the meat of the page, is a table with 8 columns - name, location, prison no, Info, Activity, Other activities, Attended and Not attended. The values in the Name column, are the offender name hyperlinked to their profile. The values in the Attended and Not attended columns are radio buttons. The rest is text. If you were to scroll to the bottom of the page there is another Print list green button.
Phew, that was a lot. This is a busy page. Let me give you some context. Most prisoners attend paid activities while in prison. This is good for their rehabilitation as it somewhat imitates life outside. In order to get paid for an activity, an offender needs to be marked with the appropriate attendance record. But before they can even make it to the activity, they need to be released from their cell and escorted to wherever they’re supposed to be. These lists are used by prison staff to unlock the right doors - location in the table, and to escort the right prisoners (name, prison no) to the right activity (activity, other activities). They print the list, because they don’t have other means of taking it on their persons while walking the corridors and unlocking cells.
Staff at the activity locations in turn mark attended or not and provide additional information as needed. The bulk action buttons “attend all” and “not require all” make it possible to tackle a large number of mundane actions at once and then fix the few outliers manually. The number of prisoners and the number of sessions allow for a quick gauge if anyone is missing or if something’s gone wrong. The date and period filters do what they say on the tin.
This page did not always look like this. Over time it has grown. Users have requested more and more information in one place. Slow internet, remember? The less they need to move between pages, the better.
This works. It is live in over 30 prisons and has been estimated to have saved 1000s of hours of staff time. Don’t ask me how they estimated it. Be like me and just believe it. Overall it has been a great success.
I have yet to explain the Reload page link. It doesn’t quite reload the page. It refreshes the state. Because this page is entirely built in React, and because it has grown such that by the time we realised we’d broken basic browser functionality we were too far gone to fix it easily.
And with this we touch on a pain point that is relatively new to us - React, it turns out, is not quite the right technology for our users. And this time they do care. I mentioned earlier that they use a remote desktop to access our sites. The whole client side rendering business does not work as intended. Worse than that, users are getting “Something is making your browser slow” alerts. You can’t do that in a high security high risk environment like a prison. You’re causing stress and concern to people who know enough to worry about such alerts. Using React is actually damaging our users’ experience because of the hardware and software limitations within which they work. We’re working to remove React from our sites. We’re in a halfway house right now - some React, some pure server side, but we’re making progress. This page will probably be our final battle.
How did we know to build it like this though? By talking to our users. We have an almost captive audience and they are engaged. We have ex-prison staff working as part of product teams and user researchers and support staff scouring the country, prodding and testing. And we have a very passionate design team. The development and the design teams are in an almost constant tug-o-war trying to minimise complexity while creating solutions that work for all use cases. It’s a friendly tug-o-war. Systematising how 135 institutions interpret policy is not easy and you can’t always do it. We value healthy push-back and provide as much support as we can.
And we’re agile. We’re almost too agile - this space is not used to rapid change and we’re doing and changing things all the time. This has caused some issues in the past and undoubtedly will in the future. While we do our best to talk to as many people as we can, we inevitably miss someone and then they come back to us with a question or a support request. On a few occasions they’ve come back to us in a panic - suddenly we’re showing something that is a security risk in their establishment, for example. This is a very big deal. It’s why having nimble deployment processes is absolutely crucial. We can revert a change and have that deployed to live in under an hour if need be. Don’t ask how I know the time. We don’t talk about that day. But we can, and we know we can because we have been through it, and that gives us confidence. We still try really hard not to break things, but there’s a certain level of uncertainty in everything we do which we have learned to live with and account for.
With all of this being said, almost every person who works in prisons and has had a chance to use our systems is pretty much in love with them. This may be the most challenging environment for web apps I’ve worked in, but it is also the most overwhelmingly rewarding one. Prison staff are used to being neglected. Nobody likes thinking about prisons, and that extends to them. So when they see someone building something for them, taking their opinions and worries into account as much as possible and really trying? Some of them cry. Some of them leap with joy. Some of them (very few indeed) look at us like we’re crazy and refuse to engage. They don’t trust the web when it doesn’t look complicated as that’s all they know. It’s truly extraordinary what a simple web form can do to people.
To wrap this up, I’d like to bring us back to the title of this talk “Building software for prisons”. The building software bit is fairly standard, as I’ve hopefully shown - there’s some code, it goes through some checks, then it goes into the cloud and all is well. The prisons bit is where the interesting challenges lie. So, how DO we go about building software for prisons? Let’s start by caring. Thank you!
]]>When a file is uploaded to Django, using the provided upload handlers, a TemporaryUploadedFile is created. It contains the file name, but not the file path. The information about the full directory tree is discarded by the upload handler.
The client has that information, so the only missing bit is how do we send it over to the server for processing. I ended up with a hidden form field called directories
which is just a simple text field in the backend. Whenever there is a change in the directory upload field, a bit of JS fires up and populates the directories
field with a JSON string representation of the directory tree where each file name is the key and its full path is the value. This is then submitted to the server as part of the upload form submission and can be processed. In my case I used it to create an archive of the upload that has the same structure as the uploaded folder.
<form method='POST' enctype="multipart/form-data">
<input type="file" id="file_input" name="file_field" webkitdirectory directory/>
<input type="text" id="directories" name="directories" hidden />
<input type="submit"/>
</form>
<script>
files = document.querySelector("#file_input").files;
document.querySelector("#file_input").addEventListener("change", function() {
files = document.querySelector("#file_input").files;
var directories = {}
for (var file of files) {
file.webkitRelativePath
directories[file.name] = file.webkitRelativePath
}
directories = JSON.stringify(directories);
document.querySelector("#directories").value = directories
});
</script>
There are some examples out there of mocking certain interactions of the Twitter client, but I couldn’t find anything concerning search. I was trying to test a controller which only cared that it god a list of objects that responded to the text
and uri
methods, it didn’t know or care about the client or any interaction with Twitter.
Stubbing the Twitter client seemed like too much effort and not something that should belong in the controller test. After some trial and error I settled on the following approach.
The controller’s only care is to serve some kind of array to the view:
class TweetsController < ActionController::Base
layout 'application'
def search
@results = TwitterSearch.new.search(params[:q])
render "tweets"
end
end
The Twitter search class deals with the actual communication with Twitter and handling of exceptions:
class TwitterSearch
attr_reader :query
def client
@client ||= Twitter::REST::Client.new do |config|
config.consumer_key = ENV["CONSUMER_KEY"]
config.consumer_secret = ENV["CONSUMER_SECRET"]
config.access_token = ENV["ACCESS_TOKEN"]
config.access_token_secret = ENV["ACCESS_SECRET"]
end
end
def search(query)
begin
results = client.search("#{query}", result_type: "recent").take(3).collect
rescue
results = []
end
end
end
For completeness, the view looks like this:
h1 Tweets
.container
= @results.each do |tweet|
p
== tweet.text
a href=(tweet.uri)
== tweet.uri
Finally, this is the test that I ended up with:
require 'test_helper'
require 'minitest/mock'
class TweetsControllerTest < ActionDispatch::IntegrationTest
test "should contain search parameter" do
Tweet = Struct.new(:text, :uri)
test_tweet = Tweet.new("This is a test tweet", "example.com")
twitter_search_stub = MiniTest::Mock.new
twitter_search_stub.expect :search, [test_tweet], ["testquery"]
TwitterSearch.stub :new, twitter_search_stub do
get "/tweets/search", params: {q: "testquery"}
assert_response :success
assert_select "p", "This is a test tweet"
assert_select "a", "example.com"
end
end
end
Let’s break that down a bit.
Tweet = Struct.new(:text, :uri)
test_tweet = Tweet.new("This is a test tweet", "example.com")
The view expects an object with the text and uri attributes. There is no Tweet class in the system, so for the purposes of the test we create one using Struct.
twitter_search_stub = MiniTest::Mock.new
twitter_search_stub.expect :search, [test_tweet], ["testquery"]
Create a mock that expects to respond to the search method, to return an array of tweets and to take an argument of “testquery”.
TwitterSearch.stub :new, twitter_search_stub do
get "/tweets/search", params: {q: "testquery"}
assert_response :success
assert_select "p", "This is a test tweet"
assert_select "a", "example.com"
end
Use the mock stub out the TwitterSearch class for everything that happeds withing the block.
There we go, mocked out Twitter search in Minitest.
]]>