Our new secret weapon for productiveness

Right this moment we’re excited to announce that we’ve teamed up with Quarto to present nbdev superpowers. nbdev provides Python programmers a typical set of instruments for utilizing Jupyter notebooks to:

• Write & distribute software program packages
• Check code, and
• Writer documentation and technical articles

Though notebooks are already extensively used for once-off exploratory work, it’s much less well-known that they’re completely able to writing high quality software program. In reality, we’ve used nbdev for a variety of software program initiatives during the last three years, together with deep studying libraries, API purchasers, Python language extensions, terminal consumer interfaces, and extra. We found that it isn’t solely able to writing nice software program however that it has additionally elevated our productiveness by 300% or extra. With nbdev, builders merely write notebooks with light-weight markup and get high-quality documentation, assessments, steady integration, and packaging without cost! Nbdev has allowed us to take care of and scale manyopen supply initiatives. Pull requests are sometimes accompanied by detailed documentation and assessments–contributors merely write notebooks.

Because of this we’re excited to share nbdev v2. It’s rewritten from the bottom up, with much-anticipated options together with:

• Interoperation with non-nbdev codebases for duties like documentation
• Assist for any static website generator
• Vast number of output mediums reminiscent of blogs, papers, slides, and web sites
• A quicker Jupyter kernel, which additionally means quicker assessments
• Cleaner and extra extensible API, which helps customized directives, customized module exporters, and extra

nbdev in trade

We have now piloted nbdev at a number of firms. We have been delighted to obtain the next suggestions, which inserts our personal expertise utilizing and creating nbdev:

David Berg, on utilizing nbdev for inner documentation at Netflix: “Previous to utilizing nbdev, documentation was probably the most cumbersome facet of our software program improvement course of… Utilizing nbdev permits us to spend extra time creating wealthy prose across the many code snippets guaranteeing the entire expertise is strong. nbdev has turned what was as soon as a chore right into a pure extension of the notebook-based testing we have been already doing.”

Erik Gaasedelen, on utilizing nbdev in manufacturing at Lyft: “I exploit this in manufacturing at my firm. It’s an superior device… nbdev streamlines every part so I can write docs, assessments, and code multi function place… The packaging can also be rather well thought out. From my perspective it’s near a Pareto enchancment over conventional Python library improvement.”

Hugo Bowne-Anderson, on utilizing nbdev for Outerbounds: “nbdev has remodeled the way in which we write documentation. Gone are the times of worrying about damaged code examples when our API modifications or [due to] human errors related to copying & pasting code into markdown information. The authoring expertise of nbdev… [allows] us to jot down prose and dwell code in a unified interface, which permits extra experimentation… On high of this, nbdev permits us to incorporate unit assessments in our documentation which mitigates the burden of sustaining the docs over time.”

Roxanna Pourzand, on utilizing nbdev for Remodel: “We’re so enthusiastic about utilizing nbdev. Our product is technical so our ensuing documentation consists of plenty of code-based examples. Earlier than nbdev, we had no approach of sustaining our code examples and guaranteeing that it was up-to-date for each command inputs and outputs. It was all guide. With nbdev, we now have this below management in a sustainable approach. Since we’ve deployed these docs, we additionally had a state of affairs the place we have been in a position to establish a bug in one in all our interfaces, which we discovered by seeing the error that was output within the documentation.”

What’s nbdev?

Nbdev embraces the dynamic nature of python and REPL-driven improvement in ways in which conventional IDEs and software program improvement workflows can’t. We totally mentioned the motivation, historical past, and targets of nbdev on this preliminary launch submit three years in the past. The creator of Jupyter, Fernando Pérez, advised us:

[Nbdev] needs to be celebrated and used much more – I’ve saved a tab along with your authentic nbdev weblog submit open for months in Chrome due to how typically I discuss with it and level others to this work

Briefly, nbdev embraces concepts from literate programming and exploratory programming. These paradigms have been revisited in platforms like XCode Playgrounds and languages like Smalltalk, LISP, and Mathematica. With nbdev, we sought to push these paradigms even additional by enabling it for one of the in style dynamic programming languages on the earth: Python.

Despite the fact that nbdev is most generally utilized in scientific computing communities as a result of its integration with Jupyter Notebooks, we’ve discovered that nbdev is properly suited to a a lot wider vary of software program. We have now used nbdev to jot down deep studying libraries, API purchasers, python language extensions,terminal consumer interfaces, and extra!

Hamel: After I use nbdev, my colleagues are sometimes astounded by how rapidly I can create and distribute high-quality python packages. I think about nbdev to be a superpower that permits me to create assessments and documentation with none extra friction, which makes all of my initiatives extra maintainable. I additionally discover writing software program with nbdev to be extra enjoyable and productive as I can iterate very quick on concepts relative to extra conventional software program engineering workflows. Lastly, with nbdev I also can use conventional text-based IDEs if I need to, so I get the most effective of each worlds.

What we discovered after three years of utilizing nbdev

Whereas nbdev was initially developed to simplify the software program improvement workflow for numerous quick.ai initiatives, we discovered that customers wished to increase nbdev to:

• Write and publish weblog posts, books, papers, and different forms of paperwork with Jupyter Notebooks
• Doc present codebases not written in nbdev
• Accommodate conventional Python conventions–for these constrained in how their code is organized and formatted
• Publish content material utilizing any static website generator

Whereas we created initiatives reminiscent of fastpages and fastdoc to perform a few of these duties, we realized that it could be higher to have a single set of versatile instruments to perform all of them. To this finish, we have been extraordinarily excited to find Quarto, an open-source technical publishing system constructed on pandoc.

Hamel: The extra I used nbdev for creating Python modules, the extra I wished to make use of it for writing blogs and documenting present codebases. The flexibility to customise the way in which notebooks are rendered (hiding vs. displaying cells, stripping output, and so forth.), together with the amenities for together with unit assessments, made it my go-to authoring device for all technical content material. I’m excited that nbdev2 unlocks all of those potentialities for everybody!

Enter Quarto: A pandoc super-processor

Quarto is a challenge that permits technical publishing with help for Jupyter Pocket book, VSCode, Observable, and plaintext editors. Moreover, Quarto permits the publishing of high-quality articles, stories, web sites, and blogs in HTML, PDF, ePub, PowerPoint slides, and extra. Quarto is maintained by RStudio, an organization with a protracted historical past of merchandise supporting literate programming, reminiscent of RMarkdown and RStudio.

Quarto is constructed on high of Pandoc, a common doc converter that helps almost any format you may consider. Pandoc achieves this seemingly magical feat by representing paperwork in a typical summary syntax tree (AST) that serves because the medium by means of which completely different codecs may be translated. By extension, Quarto permits you to generate content material in virtually any format you want! You should use pandoc filters to change the AST and the output format, which lets you use any static website generator you need, and programmatically modify and generate content material.

Quarto permits you to compose pandoc filters in a processing pipeline and apply them to particular paperwork or complete initiatives. You may as well distribute filters as Quarto extensions, which makes Quarto extraordinarily customizable.

We additionally discover Quarto compelling as a result of consumer interfaces reminiscent of remark directives (feedback that begin with #|) correlate with nbdev. In reality, we even discovered that nbdev impressed Quarto on this regard! Usually, Quarto and nbdev share many targets, and the Quarto crew has been extremely aware of our strategies. For instance, the power to create pocket book filters to change notebooks earlier than rendering. Under is a screenshot of a Jupyter pocket book rendered with Quarto and nbdev.

Lastly, Quarto helps extra programming languages than simply Python and has been including new options and fixing bugs at a powerful pace. This offers us confidence that we can increase nbdev to help extra use instances sooner or later. We focus on a few of these future instructions within the closing part.

A blazing quick pocket book kernel: execnb

A core element of nbdev is executing and testing notebooks programmatically. It is necessary that this pocket book runner executes with minimal overhead to take care of our aim of offering a pleasant developer expertise. Because of this we constructed execnb, a light-weight pocket book runner for Python kernels, which executes notebooks blazingly quick. Moreover, execnb permits parameterized execution of notebooks.

Hamel: I’ve been an enthusiastic consumer of instruments like papermill that programmatically run notebooks for use-cases like creating dashboards or enabling new sorts of machine studying workflows. I imagine execnb unlocks much more potentialities with its means to inject arbitrary code at anywhere in a pocket book, in addition to the power to go callbacks that run earlier than and/or after cells are executed. This opens up potentialities to create new forms of workflows with notebooks that I’m enthusiastic about exploring within the close to future.

In direction of a dialect of python that embraces its dynamic nature

One technique to perceive nbdev is a part of an ecosystem that’s designed to embrace Python’s dynamic properties for REPL-driven software program engineering. Just like Clojure, our aim is to supply instruments that take away all friction from utilizing the REPL in your programming workflow. We imagine that the REPL enhances developer workflows due to context-sensitive auto-completion, signature inspection, and documentation–all based mostly on the precise state of your code, and none of which can be found in IDEs that rely solely on static evaluation. We have now discovered that for that reason, nbdev, with its Jupyter pocket book basis, makes programming considerably extra productive and pleasing.

Our efforts to help REPL-driven improvement and literate programming should not restricted to nbdev. We preserve a lot of libraries that reach python to bolster this programming expertise. Probably the most notable of those libraries is fastcore, which extends Python when it comes to testing, documenting code, metaprogramming, attribute helpers, enhanced representations of objects, and notebook-friendly patching. This weblog submit provides a mild introduction to fastcore. Along with literate programming, fastcore encourages conventions reminiscent of brevity and environment friendly use of vertical area so you may accomplish extra with considerably much less code. For instance, beneath is a straightforward decorator that permits notebook-friendly patching:

We imagine that this mixture of a brand new developer workflow (nbdev), Python extensions (fastcore), and related norms kind a brand new dialect of Python that’s centered on leveraging its dynamic nature–in distinction to an ever-growing pattern towards static evaluation. We suspect that this dialect of Python can be extra productive for programmers in lots of situations. We’re framing this ecosystem as a “dialect” as it’s nonetheless very a lot Python and is approachable by anybody who’s accustomed to the language. Moreover, regardless of nbdev’s pocket book workflow, our instruments generate plaintext modules that may be navigated and edited with text-based IDEs, permitting programmers to expertise the most effective of each worlds, in the event that they need.

Hamel: I imagine this framing of a Python dialect is essential to correctly understanding what nbdev is. Whereas it might be tempting to get caught on particular options or technical particulars of nbdev, it’s helpful to zoom out to grasp the general intent of making a greater workflow fairly than conforming too rigidly to present ones. A very good analogy is TypeScript’s relationship with JavaScript: it’s an extension of an present programming language that helps a brand new approach of programming. I encourage you to deal with nbdev similarly: be prepared to attempt new methods of programming and observe which tradeoffs resonate with you. On the very least, I imagine nbdev is a enjoyable technique to expertise a special approach of writing software program, which is able to broaden your horizons about programming basically, all with out having to be taught a completely new programming language!

The way forward for nbdev

Whereas we’re enthusiastic about nbdev2, we imagine now we have solely scratched the floor of what’s attainable. We’re contemplating the next options:

• Supporting extra languages past Python, reminiscent of Julia, R and JavaScript
• Providing interfaces for executing parameterized notebooks that mimic Python scripts
• Extensions for extra static website mills and filters
• Supporting alternate testing backends, reminiscent of pytest
• Supporting a larger variety of docstring codecs, reminiscent of Google-style docstrings
• Extra choices to make use of plain-text or human readable pocket book backends aside from JSON

You probably have fascinating concepts about how nbdev may be prolonged, please drop and chat with us on discord or submit a message within the boards.

How one can get began with nbdev

Our challenge’s web site is at nbdev.quick.ai, the place we can be posting tutorials, examples, and extra documentation within the coming days.

Thank You

This new model of nbdev was a crew effort by many fantastic individuals. We need to spotlight two individuals who have made excellent contributions:

• Wasim Lorgat was instrumental throughout completely different areas, together with vital contributions to fastcore, execnb, and nbdev, in addition to the implementation of the brand new nbdev residence web page. With Wasim’s assist, we have been in a position to push nbdev to a brand new degree of performance and high quality.

• JJ Allaire will not be solely the CEO of RStudio but in addition the steward of Quarto. JJ was extremely responsive and desirous to work with us on nbdev and added many options to Quarto particularly with nbdev in thoughts, reminiscent of pocket book filters. We have been additionally astounded by the eye to element and the tempo at which bugs are addressed. This new model of nbdev wouldn’t have been attainable with out JJ’s assist, and we’re excited to proceed to work with him.

We additionally need to thank the wonderful fastai group, notably Isaac Flath, Benjamin Warner and Zach Mueller for his or her tireless work on this challenge.

A dialog with JJ Allaire

To have a good time the launch of nbdev v2 and Quarto, Jeremy sat down with the CEO of Posit (beforehand generally known as RStudio, the corporate behind Quarto), JJ Allaire, to speak about software program improvement, scientific publishing, R, Python, literate programming, and rather more.