David Dickson About me

Operating as a Distributed Team

At Sage AI Labs (SAIL) we have talented Engineers and Data Scientists spread across North America, Europe, Australia and often have to collaborate with teams in Asia. As a result the group has made a concerted effort to adopt techniques and processes that help us succeed as a distributed team.

Distributed team members are not co-located and thus rarely see each other, this differs from a remote team that still works in the same office but in a different location to a company’s headquarters.

Some benefits of a distributed team

Hiring top talent

With a distributed team as you open yourself up to a talent pool that extends beyond geographic boundaries.

Diverse team

Cultural diversity plays a critical role when it comes to thinking out of the box with a number of studies showing that diversity helps bring out unconventional solutions and new ideas. Having team members spread across the globe is a great way to foster such diversity.

Productivity

As team members are given autonomy to complete their work and control over their environment they can set up a work flow and workplace that best suits them, allowing them to optimize for productivity.

It feels like the future

When I talk to others about the location of my team and how we are approaching distributed work excites me. There is little doubt that the nature of work is changing and to be at the forefront of this change is professionally satisfying.

Take note

Often companies start hiring distributed workers and assume that their current processes will transfer into this new dynamic. This is a flawed assumption yet few organizations make the effort to reorient themselves. Below list some of the practices SAIL has adopted while transitioning into a distributed team.

SAIL’s distributed team practices

Taking inspiration from companies such as GitHub that have pioneered distributed teams, speaking to peers within our network that run distributed workforces, and iterating on our own approach we came up with a set of conventions. Below lists some of these key practices SAIL has adopted while evolving as a distributed team.

Meetings

If one of us is remote in a meeting, all should be on their laptop with their camera turned on. This removes the ability for those congregated in a meeting room to dismiss those remote and creates a level playing feed for discussion. Plan the agenda carefully and share it in advance.

If something complex is to be discussed, link to a document with the details of the meeting so that discussion can happen asynchronously prior to the meeting (literally getting people on the same page).

Distributed champion

We have a single person responsible for championing our distributed culture and approach. We also have periodic retrospectives run by our champion so that we can reflect on how we might improve our distributed processes. They also serve as a contact that other team members can channel potential ideas on how we might improve.

Work from home policy

So as to build empathy with remote work we encourage everyone to work remotely at least once a week, even if they have access to an office they can reside in.

Over communicate

As a team we are encouraged to be mindful of the fact that at any one point in time only half of our team is available so it is critical to over-communicate important things that are discussed. We try to radiate decisions on the Internet either via broadcasting them in appropriate Slack channels or other forms to catalog what has been decided.

I started writing this post well before the COVID-19 pandemic arrived however as remote working is increasingly seen as a necessity to practice social distancing I felt compelled to finish my post. I am passionate about distributed work and am sure it will very quickly become the new normal as such I hope some of these ideas resonate with teams as they transition.

Compass Re-architecture

Early this year at Compass we decided to significantly redesign our core product with a focus on making it self-service. This juncture was the perfect opportunity for us to also evaluate the technical stack we were leveraging. This evaluation lead to some specific changes as outlined below.

  New Stack Old Stack
Database PostgreSQL MongoDB
Inter-Process Communication RabbitMQ RabbitMQ
Web Backend Sinatra Rails
Web Frontend AngularJS AngularJS

It is now 6 months from our re-architecture and I wanted to discuss the outcomes from our move.

MongoDB to PostgreSQL

The problem we were finding with Mongo was that our data model was becoming a bit of a mess. As a reporting and benchmarking tool it is fundamental that Compass maintains a well structured database. One of the fundamental features of MongoDB (schemaless storage) was proving to be a significant pain point.

Having our storage engine define our schema has certainly been a positive decision. We often run algorithms over very flat data structures - as a result we found a relational data model actually mapped effectively to our problem space. Although there are certainly domains where persisting documents can increase productivity, this simply wasn’t the case for us.

Compass is made up of both Software Engineers and Data Scientists. Much of a Data Scientist’s job is often spent cleaning and analyzing data. Moving to PostgreSQL has made it significantly easier for our data science team to explore critical data. The improved consistency coupled with their comfortability with SQL has lead to more efficient turn around in problem understanding and feature specification.

As PostgreSQL comes equipped with some NoSQL features our move really has felt like the best of both worlds. With PostgreSQL you can get document database-like capabilities using the JSON data type, and key/value storage with the HSTORE module. As a result we have been able to make use of these features where appropriate.

Rails to Sinatra

The biggest problem with our existing Rails application was that it didn’t have a clear separation of concerns. At times the web layer was implemented as simple restful endpoints, at other times it was rendering complex views. With our move to Sinatra we wanted to address this inconsistency and have a clear separation between the client (Angular SPA) and our web API layer. Furthermore in building a single page application a lot of what Rails would provide is overkill.

Since moving to Sinatra we have found it excellent for building out a web API layer. Its flexibility gave us the control over what libraries we leverage in building out our application we wanted. Furthermore its lightweight nature and simple method for specifying routes has given the team a restful focus, encouraging the separation between web and client we were striving for.

One interesting positive side effect that we didn’t anticipate, is that leveraging a micro framework has helped our junior developers better understand web semantics. In having less ‘configuration magic’, ramp up time for our junior developers may be greater, however it has forced them to gain a deeper understanding of how web applications are built.

Conclusion

Six months on from our architectural changes and the team remains busy building out new features for Compass. Software is forever changing and I am sure our stack will have significant evolutions in the years to come. But as it stands the team remains happy with the architectural decisions made and is excited about evolving the platform. Moving to Sinatra gave us the clear separation between web and client we were looking for, and PostgreSQL the tighter structure and subsequent boost in productivity we needed.