I know this might come as a shock to you, but most programming job applicants suck. I’ve interviewed my fair share this month, and it’ll be a lot easier for all of us if I tell you upfront what I’m looking for.

As a hiring manager, my job is to make sure you can do the job you’re applying for. For programming that means you need to be able to program. So when I whip out a laptop in our round-one interview and ask you to write some code, try to hide your terrified expression.

In the last week of April, nearly 23 percent of all traffic to news sites tracked by web analytics firm Parse.ly came from search engines. Google alone accounts for nearly half of external referral traffic—traffic, that is, that comes from platforms, apps, and other outside sources— to news sites. Together with the fact that Facebook […]

Let’s start by summing up this release with an example.

Comprehensive Example

Here’s a nice showcase of DifferentialEquations.jl: Neural ODE with batching on
the GPU (without internal data transfers) with high order adaptive implicit ODE
solvers for stiff equations using matrix-free Newton-Krylov via preconditioned
GMRES and trained using checkpointed adjoint equations. Few programs work
directly with neural networks and allow for batching, few utilize GPUs, few
have methods applicable to highly stiff equations, few allow for large stiff
equations via matrix-free Newton-Krylov, and finally few have checkpointed
adjoints. This is all done in a high level programming language. What does the
code for this look like?

using OrdinaryDiffEq, Flux, DiffEqFlux, DiffEqOperators, CuArrays
x = Float32[2.; 0.]|>gpu
tspan = Float32.((0.0f0,25.0f0))
dudt = Chain(Dense(2,50,tanh),Dense(50,2))|>gpu
p = DiffEqFlux.destructure(dudt)
dudt_(du,u::TrackedArray,p,t) = u .= DiffEqFlux.restructure(dudt,p)(u)
dudt_(du,u::AbstractArray,p,t) = u .= Flux.data(DiffEqFlux.restructure(dudt,p)(u))
ff = ODEFunction(dudt_,jac_prototype = JacVecOperator(dudt_,x))
prob = ODEProblem(ff,x,tspan,p)
diffeq_adjoint(p,prob,KenCarp4(linsolve=LinSolveGMRES());u0=x,
saveat=0.0:0.1:25.0,backsolve=false)

That is 10 lines of code, and we can continue to make it even more succinct.

Now, onto the release highlights.

Full GPU Support in ODE Solvers

Now not just the non-stiff ODE solvers but the stiff ODE solvers allow for
the initial condition to be a GPUArray, with the internal methods not
performing any indexing in order to allow for all computations to take place
on the GPU without data transfers. This allows for expensive right-hand side
calculations, like those in neural ODEs or PDE discretizations, to utilize
GPU acceleration without worrying about whether the cost of data
transfers will overtake the solver speed enhancements.

While the presence of broadcast throughout the solvers might worry one about
performance…

Fast DiffEq-Specific Broadcast

Yingbo Ma (@YingboMa) implemented a fancy broadcast wrapper that allows for
all sorts of information to be passed to the compiler in the differential
equation solver’s internals, making a bunch of no-aliasing and sizing assumptions
that are normally not possible. These change the internals to all use a
special @.. which turns out to be faster than standard loops, and this is the
magic that really enabled the GPU support to happen without performance
regressions (and in fact, we got some speedups from this, close to 2x in some
cases!)

Smart linsolve defaults and LinSolveGMRES

One of the biggest performance-based features to be released is smarter linsolve
defaults. If you are using dense arrays with a standard Julia build, OpenBLAS
does not perform recursive LU factorizations which we found to be suboptimal
by about 5x in some cases. Thus our default linear solver now automatically
detects the BLAS installation and utilizes RecursiveFactorizations.jl to give
this speedup for many standard stiff ODE cases. In addition, if you passed a
sparse Jacobian for the jac_prototype, the linear solver now automatically
switches to a form that works for sparse Jacobians. If you use an
AbstractDiffEqOperator, the default linear solver automatically switches to
a Krylov subspace method (GMRES) and utilizes the matrix-free operator directly.
Banded matrices and Jacobians on the GPU are now automatically handled as well.

Of course, that’s just the defaults, and most of this was possible before but
now has just been made more accessible. In addition to these, the ability to
easily switch to GMRES was added via LinSolveGMRES. Just add
linsolve = LinSolveGMRES() to any native Julia algorithm with a swappable
linear solver and it’ll switch to using GMRES. In this you can pass options
for preconditioners and tolerances as well. We will continue to integrate this
better into our integrators as doing so will enhance the efficiency when
solving large sparse systems.

Automated J*v Products via Autodifferentiation

When using GMRES, one does not need to construct the full Jacobian matrix.
Instead, one can simply use the directional derivatives in the direction of
v in order to compute J*v. This has now been put into an operator form
via JacVecOperator(dudt_,x), so now users can directly ask for this to
occur using one line. It allows for the use of autodifferentiation or
numerical differentiation to calculate the J*v.

DEStats

One of the nichest but nicest new features is DEStats. If you do sol.destats
then you will see a load of information on how many steps were taken, how many
f calls were done, etc. giving a broad overview of the performance of the
algorithm. Thanks to Kanav Gupta (@kanav99) and Yingbo Ma (@YingboMa) for really
driving this feature since it has allowed for a lot of these optimizations to
be more thoroughly investigated. You can expect DiffEq development to
accelerate with this information!

Improved Jacobian Reuse

One of the things which was noticed using DEStats was that the amount of Jacobians
and inversions that were being calculated could be severly reduced. Yingbo Ma (@YingboMa)
did just that, greatly increasing the performance of all implicit methods like
KenCarp4 showing cases in the 1000+ range where OrdinaryDiffEq’s native
methods outperformed Sundials CVODE_BDF. This still has plenty of room for
improvement.

DiffEqBiological performance improvements for large networks (speed and sparsity)

Samuel Isaacson (@isaacson) has been instrumental in improving DiffEqBiological.jl
and its ability to handle large reaction networks. It can now parse the networks
much faster and can build Jacobians which utilize sparse matrices. It pairs
with his ParseRxns(???) library and has been a major source of large stiff
test problems!

Partial Neural ODEs, Batching and GPU Fixes

We now have working examples of partial neural differential equations, which
are equations which have pre-specified portions that are known while others
are learnable neural networks. These also allow for batched data and GPU
acceleration. Not much else to say except let your neural diffeqs go wild!

Low Memory RK Optimality and Alias_u0

Kanav Gupta (@kanav99) and Hendrik Ranocha (@ranocha) did amazing jobs at doing memory optimizations of
low-memory Runge-Kutta methods for hyperbolic or advection-dominated PDEs.
Essentially these methods have a minimal number of registers which are
theoretically required for the method. Kanav added some tricks to the implementation
(using a fun = -> += overload idea) and Henrick added the alias_u0 argument
to allow for using the passed in initial condition as one of the registers. Unit
tests confirm that our implementations achieve the minimum possible number of
registers, allowing for large PDE discretizations to make use of
DifferentialEquations.jl without loss of memory efficiency. We hope to see
this in use in some large-scale simulation software!

More Robust Callbacks

Our ContinuousCallback implementation now has increased robustness in double
event detection, using a new strategy. Try to break it.

GBS Extrapolation

New contributor Konstantin Althaus (@AlthausKonstantin) implemented midpoint
extrapolation methods for ODEs using Barycentric formulas and different a
daptivity behaviors. We will be investigating these methods for their
parallelizability via multithreading in the context of stiff and non-stiff ODEs.

ModelingToolkit.jl Release

ModelingToolkit.jl has now gotten some form of a stable release. A lot of credit
goes to Harrison Grodin (@HarrisonGrodin). While it has
already been out there and found quite a bit of use, it has really picked up
steam over the last year as a modeling framework suitable for the flexibility
DifferentialEquations.jl. We hope to continue its development and add features
like event handling to its IR.

SUNDIALS J*v interface, stats, and preconditioners

While we are phasing out Sundials from our standard DifferentialEquations.jl
practice, the Sundials.jl continues to improve as we add more features to
benchmark against. Sundials’ J*v interface has now been exposed, so adding a
DiffEqOperator to the jac_prototype will work with Sundials. DEStats is
hooked up to Sundials, and now you can pass preconditioners to its internal
Newton-Krylov methods.

Next Directions

Improved nonlinear solvers for stiff SDE handling
More adaptive methods for SDEs
Better boundary condition handling in DiffEqOperators.jl
More native implicit ODE (DAE) solvers
Adaptivity in the MIRK BVP solvers
LSODA integrator interface
Improved BDF

Stripe will hire over a hundred remote engineers this year. They will be deployed across every major engineering workstream at Stripe.

## Our users are everywhere. We have to be, too.

Our remotes keep us close to our customers, which is key to building great products. They are deeply embedded in the rhythms of their cities. They see how people purchase food differently in bodegas, konbini, and darshinis. They know why it is important to engineer robustness in the face of slow, unreliable internet connections. They have worked in and run businesses that don’t have access to global payments infrastructure.

Stripe has had hundreds of extremely high-impact remote employees since inception. Historically, they’ve reported into teams based in one of our hubs. We had a strong preference for managers to be located in-office and for teams to be office-centric, to maximize face-to-face bandwidth when doing creative work.

As we have grown as a company, we have learned some things.

One is that the technological substrate of collaboration has gotten *shockingly* good over the last decade. Most engineering work at Stripe happens in conversations between engineers, quiet thinking, and turning those thoughts into artifacts. Of these, thinking is the only one that doesn’t primarily happen online.

There was a time when writing on a whiteboard had substantially higher bandwidth than a Word doc over email. Thankfully Google Docs, Slack, git, Zoom, and the like deliver high-bandwidth synchronous collaboration on creative work. The experience of using them is so remarkably good that we only notice it when something is broken. Since you write code via pull requests and not whiteboards, your reviewer needs to have access to the same PR; having access to the same whiteboard is strictly optional.

While we did not initially plan to make hiring remotes a huge part of our engineering efforts, our remote employees have outperformed all expectations. Foundational elements of the Stripe technology stack, our products, our business, and our culture were contributed by remotes. We would be a greatly diminished company without them.

## Stripe’s new remote engineering hub

We have seen such promising results from our remote engineers that we are greatly increasing our investment in remote engineering.

**We are formalizing our Remote engineering hub.** It is coequal with our physical hubs, and will benefit from some of our experience in
[scaling engineering
organizations](https://stripe.com/atlas/guides/scaling-eng).
For example, there will be dedicated engineering teams in the Remote hub that exist in no other hub. (Some individuals report to a team located in a different hub, and we expect this will remain common, but the bulk of high-bandwidth coworker relationships are within-hub.) We also have a remote engineering lead, analogous to the site leads we have for our physical hubs.

**We are expanding the scope we will hire for remotely**. In addition to hiring engineers, we plan to begin hiring remote product managers, engineering managers, and technical program managers later this year. (We will continue hiring remote employees in non-engineering positions across the company as well.)

**We intend to expand our remote engineering hiring aggressively.** We will hire at least a hundred remote engineers this year. We expect to be constrained primarily by our capacity to onboard and support new remote engineers, and we will work to increase that capacity.

**We will continue to improve the experience of being a remote.** We have carefully tracked the experience of our remote employees, including in our twice-annual employee survey. Most recently, 73% of engineers at Stripe believe we do a good job of integrating remote employees.

Great user experiences are made in the tiny details. We care about the details to a degree that is borderline obsessive. A recent example: we wrote code to attach a videoconferencing link to every calendar invitation by default, so that remotes never feel awkward having to ask for one.

## More to come

There are still some constraints on our ambitions. In our first phase, we will be focused primarily on remote engineers in North America, starting with the US and Canada. While we are confident that great work is possible within close time zones, we don’t yet have structures to give remotes a reliably good experience working across large time zone differences. And though we intend to hire remote engineers in Europe and Asia eventually, our hubs in Dublin and Singapore are not sufficiently established to support remotes just yet.

Most engineers working at Stripe are full-time employees, with a full benefits suite. There is substantial organizational, legal, and financial infrastructure required to support each new jurisdiction we hire in, so we have to be measured in how quickly we expand. We can support most US states today, and plan to expand our hiring capabilities to include jurisdictions covering more than 90% of the US population as quickly as possible. We intend, over the longer term, to be everywhere our customers are.

We will continue encouraging governments worldwide to lower barriers to hiring. Our customers, from startups to international conglomerates, all feel the pain of this. We think making it easier for companies to hire would produce a step-function increase in global GDP.

## We want to talk to you

We would love to talk about our Remote hub or [remote positions at
Stripe](https://stripe.com/jobs/search?l=remote). Our
CEO and co-founder, Patrick Collison, and I will host a remote coffee on May 22, 2019;
[sign up to be invited](https://stripe.events/remote-coffee) to it. We are also, and always, available on the internet.

But recently we’ve come to realize this is not enough.