The Times They Are a-Changin’: A Data-Driven Portrait of New Trends in How We Build Software, Open Source, & What Even is "Entry-Level" Now

Much has changed in a short period of time in three areas: (1) the way we build software today in industry is very different than it was even 5 years ago, (2) the dynamics of open source have changed, (3) the average developer has drastically changed.

Using data I’ve collected from government statistics, and from various surveys and interviews, I’ll make the point that perhaps a traditional computer science degree might be overkill for average application developer jobs out there: a lot of jobs predominantly involve gluing together open source components rather than having to write components from scratch. I’ll also show that if you’ve been working for 5+ years in industry, then you are in a tiny minority of veteran programmers because there is a literal onslaught of entry-level developers entering the market, at a rate unseen before.

These data points are interesting because it affects all of us – both Academic and Industry.

Transcript

1. The Times They Are a-Changin’ A Data-Driven Portrait of New Trends in How We Build Software, Open Source, & What Even is "Entry-Level" Now Heather Miller SPLASH Rebase, October 24th 2019, Athens Greece @heathercmiller
2. A bit about me OH HAI Assistant Professor in the School of Computer Science @ CMU
3. A bit about me OH HAI Assistant Professor in the School of Computer Science @ CMU Founded the Scala Center, 2016
4. A bit about me OH HAI Assistant Professor in the School of Computer Science @ CMU Founded the Scala Center, 2016 PhD in Computer Science, 2015 under Martin Odersky
5. A bit about me OH HAI Assistant Professor in the School of Computer Science @ CMU Founded the Scala Center, 2016 PhD in Computer Science, 2015 under Martin Odersky • Scala Futures • Scala’s concurrency libraries • Typeclass derivation • Lightweight type system extensions • Programming models for distributed programming • Coursera MOOCs Worked a lot on Scala
6. A bit about me OH HAI Assistant Professor in the School of Computer Science @ CMU Joined CMU as an assistant prof in 2018 My research: - bringing programming language techniques to dist. systems - making microservice architectures more reliable - distributed actor runtimes Founded the Scala Center, 2016 PhD in Computer Science, 2015 under Martin Odersky • Scala Futures • Scala’s concurrency libraries • Typeclass derivation • Lightweight type system extensions • Programming models for distributed programming • Coursera MOOCs Worked a lot on Scala
7. Building a new lab at CMU Chris Meiklejohn @cmeik Zeeshan Lakhani @zeeshanlakhani Building and formalizing language-level distributed and concurrent programming abstractions (monotonic, serializable, convergent, dataflow models). Broadly, PL-support for distributed systems. with some fine folks! Assistant Professor in the School of Computer Science + you? We’re always looking for new students!
8. I’m the founding director. And suddenly my focus is 200% what is happening in open source Scala, and how we can keep growing our ecosystem, tools, and improve developer experience for anyone. SCALA CENTER Not only paying customers. But anyone with an internet connection. A good developer experience should be free. This shift in focus was eye-opening. I quickly observed problems with the health of some of our core projects in our own ecosystem that was cause for concern. And what’s worse, this trend is common throughout the open source community.
9. So I started collecting data on these fast-changing trends
10. Things that are changing fast: How we build software Open Source Our idea of software engineers What we actually do nowadays when we sit down to build an app. The common infrastructure and tools that we all depend on What SWEs should know, how much experience they have, and who they are. & that more people should be aware of
11. Things that are changing fast: How we build software Open Source Our idea of software engineers What we actually do nowadays when we sit down to build an app. The common infrastructure and tools that we all depend on What SWEs should know, how much experience they have, and who they are. & that more people should be aware of This talk will cover fast-changing trends in these 3 areas
12. How people are getting into tech is changing FIRST,
13. We all already know that hiring is difficult HOW PEOPLE ARE GETTING INTO TECH IS CHANGING
14. We all already know that hiring is difficult HOW PEOPLE ARE GETTING INTO TECH IS CHANGING
15. We all already know that hiring is difficult HOW PEOPLE ARE GETTING INTO TECH IS CHANGING
16. We all already know that hiring is difficult HOW PEOPLE ARE GETTING INTO TECH IS CHANGING
17. There’s a shortage of tech workers HOW PEOPLE ARE GETTING INTO TECH IS CHANGING
18. There’s a shortage of tech workers Students graduating with CS/IT-related bachelor’s degrees *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
19. There’s a shortage of tech workers Students graduating with CS/IT-related bachelor’s degrees Last available data point: 2014-2015: 60,309 degrees *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
20. There’s a shortage of tech workers Students graduating with CS/IT-related bachelor’s degrees Last available data point: 2014-2015: 60,309 degrees growth of ~7.4% per year *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
21. There’s a shortage of tech workers Students graduating with CS/IT-related bachelor’s degrees Last available data point: 2014-2015: 60,309 degrees *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
22. There’s a shortage of tech workers Students graduating with CS/IT-related bachelor’s degrees Last available data point: 2014-2015: 60,309 degrees Meanwhile, over 500,000 computing- related job openings right now! *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
23. There’s still a shortage of tech workers, when you include code bootcamps too *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
24. By 2026, there will be 3.5 million computing-related job openings* *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
25. By 2026, there will be 3.5 million computing-related job openings* It is estimated that only 19% of these jobs can be filled by US computing degree bachelor’s degree recipients. *Department of Labor Statistics, Employment Projections (Occupational Category: 15-1100) Includes new and replacement jobs and assumes current undergraduate degree (CIP 11) production levels persist https://www.ncwit.org/sites/ default/files/resources/ btn_05092019_web.pdf *note, this is a US-centric view!
26. A large portion of professional developers are new THE TECH WORKERS WE HAVE… 2017 2018 2019
27. A large portion of professional developers are new THE TECH WORKERS WE HAVE… 2017 2018 2019 50.1% have up to 5yrs of experience (20% 2yrs or less, 32% 3yrs or less)
28. A large portion of professional developers are new THE TECH WORKERS WE HAVE… 2017 2018 2019 50.1% have up to 5yrs of experience (20% 2yrs or less, 32% 3yrs or less) 57.5% have up to 5yrs of experience
29. A large portion of professional developers are new THE TECH WORKERS WE HAVE… 2017 2018 2019 50.1% have up to 5yrs of experience (20% 2yrs or less, 32% 3yrs or less) 57.5% have up to 5yrs of experience 41% have less than 5yrs of experience
30. We need to adapt, culturally, to make room for lots more newcomers The demand for developers is just going to get more and more ridiculous. TAKEAWAY: The years of experience of practicing SWEs is dropping overall. There’s a tidal wave of newcomers entering our profession, and it’s not going to slow down. It’s going to pick up speed.
31. We need to adapt, culturally, to make room for lots more newcomers The demand for developers is just going to get more and more ridiculous. TAKEAWAY: The years of experience of practicing SWEs is dropping overall. There’s a tidal wave of newcomers entering our profession, and it’s not going to slow down. It’s going to pick up speed. The New Jobs By Marina Krakovsky Communications of the ACM, January 2018, Vol. 61 No. 1, Pages 21-23 10.1145/3157077 What do we do? "New frameworks are lowering the barrier to entry," Caleb Fristoe (founder of CodeTN) says; that's a far cry from the days when you had to learn the syntax of several programming languages to build useful software. "Rather than typing these seven lines of code to get a menu to pop down, you just download the framework from a code base that allows you to do that in a simpler way," he explains. "Frameworks are taking the hard work that developers prided themselves on out of the equation." https://cacm.acm.org/magazines/2018/1/223883-the- new-jobs/fulltext
32. Existing devs are burning out TAKEAWAY: “Unable to fill tech vacancies, employers shuffle off additional duties to current employees, which leads to burnout and has a negative impact on local business development. Over 30% of respondents surveyed by Indeed admit that this issue accelerates staff turnover.” US Tech Talent Shortage in Numbers March 26, 2019 https://www.daxx.com/blog/development-trends/software- engineer-shortage-us-2019 “With companies unable to fill open positions, current employees are expected to fill the gaps. In many cases this results in employee turnover. Over a third of respondents we surveyed (36%) said the lack of timely hiring has caused burnout in existing employees and affected their businesses.” Is the Tech Talent War Hurting Innovation? Hiring Managers and Tech Recruiters Respond December 5, 2016 http://blog.indeed.com/2016/12/05/impact-of-tech-talent- shortage/
33. Obviously, increased diversity would help We know that the people who develop software are not a representative of sample of society. Making more underrepresented minorities at home in tech is an obvious solution to increasing our numbers. TAKEAWAY: Is the Tech Talent War Hurting Innovation? Hiring Managers and Tech Recruiters Respond Posted on December 5, 2016 http://blog.indeed.com/2016/12/05/impact-of-tech-talent- shortage/ Immigration = good, more tech workers Remote workers = good, more tech workers But also… *note, this is a US-centric view!
34. We need to care about diversity for more than economics alone. A QUICK ASIDE…
35. Increased diversity = increased productivity PAPER: Gender and tenure diversity in GitHub teams. Gender and tenure diversity in GitHub teams. Vasilescu, B., Posnett, D., Ray, B., Brand, M.G.J. van den, Serebrenik, A., Devanbu, P., and Filkov, V. CHI 2015 Research from one of my colleagues: There is evidence that software teams that are more diverse are more productive. Holding other confounds fixed, teams that are more diverse with respect to gender and/or tenure/ experience tend to write code faster than teams that are less diverse.
36. How do we stop people from disengaging? PAPER: The Impact of Social Capital on Sustained Participation in Open Source Going Farther Together: The Impact of Social Capital on Sustained Participation in Open Source. Qiu, H.S., Nolte, A., Brown, A., Serebrenik, A., and Vasilescu, B. ICSE 2019 Women disengage earlier than men:
37. How do we stop people from disengaging? PAPER: The Impact of Social Capital on Sustained Participation in Open Source Going Farther Together: The Impact of Social Capital on Sustained Participation in Open Source. Qiu, H.S., Nolte, A., Brown, A., Serebrenik, A., and Vasilescu, B. ICSE 2019 Women disengage earlier than men:
38. How do we stop people from disengaging? PAPER: The Impact of Social Capital on Sustained Participation in Open Source Going Farther Together: The Impact of Social Capital on Sustained Participation in Open Source. Qiu, H.S., Nolte, A., Brown, A., Serebrenik, A., and Vasilescu, B. ICSE 2019 People on informationally diverse teams engage longer: Take away: Invest in building social capital & Foster informationally diverse teams
39. A bit about how we build software FIRST,
40. Remember this? We used to have to work hard to convince companies to trust open source software.
41. Black Duck (now Synopsys) runs an annual survey asking companies about their open source use. They survey >1,000 companies about their open source usage. https://www.blackducksoftware.com/2016-future-of-open-source OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey
42. How People Build Software Changed Dramatically Since 2010 2010 39% of companies said they “ran on open source” https://www.blackducksoftware.com/2016-future-of-open-source OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey Interviewed 1,240 companies (2015) Interviewed 1,313 companies (2016)
43. How People Build Software Changed Dramatically Since 2010 2010 39% of companies said they “ran on open source” 2015 78% of companies said they “ran on open source” https://www.blackducksoftware.com/2016-future-of-open-source OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey Interviewed 1,240 companies (2015) Interviewed 1,313 companies (2016)
44. How People Build Software Changed Dramatically Since 2010 2010 39% of companies said they “ran on open source” 2015 78% of companies said they “ran on open source” https://www.blackducksoftware.com/2016-future-of-open-source OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey 2x This is up 2x over 2010! Interviewed 1,240 companies (2015) Interviewed 1,313 companies (2016)
45. How People Build Software Changed Dramatically Since 2010 2010 39% of companies said they “ran on open source” 2015 78% of companies said they “ran on open source” https://www.blackducksoftware.com/2016-future-of-open-source OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey 2x This is up 2x over 2010! Interviewed 1,240 companies (2015) Interviewed 1,313 companies (2016) COMPANIES ARE DEPENDING MORE AND MORE ON OSS!
46. Why did companies suddenly decide to shift building atop OSS? OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey https://www.blackducksoftware.com/2016-future-of-open-source
47. Why did companies suddenly decide to shift building atop OSS? OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey https://www.blackducksoftware.com/2016-future-of-open-source 2016 Top 3 reasons to use OSS: - #1 quality of solutions - #2 competitive features & technical capabilities - #3 ability to customize & fix
48. Why did companies suddenly decide to shift building atop OSS? OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey https://www.blackducksoftware.com/2016-future-of-open-source 2016 Top 3 reasons to use OSS: - #1 quality of solutions - #2 competitive features & technical capabilities - #3 ability to customize & fix 2015 OSS vs proprietary: - 66% of companies consider OSS options before proprietary alternatives
49. Why did companies suddenly decide to shift building atop OSS? OPEN SOURCE SURVEYS: 2015 & 2016 Black Duck “Future of Open Source” Survey https://www.blackducksoftware.com/2016-future-of-open-source 2016 Top 3 reasons to use OSS: - #1 quality of solutions - #2 competitive features & technical capabilities - #3 ability to customize & fix OPEN SOURCE BECAME THE DEFAULT CHOICE 2015 OSS vs proprietary: - 66% of companies consider OSS options before proprietary alternatives
50. The rapid increase in reliance on open source continues OPEN SOURCE SURVEYS: Black Duck 2017 Survey https://www.blackducksoftware.com/open-source-360deg-survey Main attributed reason: - low cost with no vendor lock-in 2017 60% of companies surveyed increased open source usage.
51. Everything is OSS now Scanned/analyzed (anonymized) data of over 1,100 commercial code bases. OPEN SOURCE SURVEYS: Synopsys 2018 Survey (was Black Duck) https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/2018-ossra.pdf
52. Everything is OSS now Scanned/analyzed (anonymized) data of over 1,100 commercial code bases. OPEN SOURCE SURVEYS: Synopsys 2018 Survey (was Black Duck) https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/2018-ossra.pdf - Open source components in 96% of applications scanned! - Average of 257 open source components per application!
53. Everything is OSS now Scanned/analyzed (anonymized) data of over 1,100 commercial code bases. OPEN SOURCE SURVEYS: Synopsys 2018 Survey (was Black Duck) https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/2018-ossra.pdf - Open source components in 96% of applications scanned! - Average of 257 open source components per application! 96% USED OSS IN 2018!
54. Everything is OSS now Scanned/analyzed (anonymized) data of over 1,100 commercial code bases. OPEN SOURCE SURVEYS: Synopsys 2018 Survey (was Black Duck) https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/2018-ossra.pdf - Open source components in 96% of applications scanned! - Average of 257 open source components per application! 96% USED OSS IN 2018! In 2017, 36% of code base was open source components. In 2018, that number is 57%.
55. Everything is OSS now Scanned/analyzed (anonymized) data of over 1,100 commercial code bases. OPEN SOURCE SURVEYS: Synopsys 2018 Survey (was Black Duck) https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/2018-ossra.pdf - Open source components in 96% of applications scanned! - Average of 257 open source components per application! 96% USED OSS IN 2018! In 2017, 36% of code base was open source components. In 2018, that number is 57%. MANY APPS ARE NOW MORE OPEN SOURCE CODE THAN PROPRIETARY
56. Software now is mostly made out of OSS components OPEN SOURCE SURVEYS: 2018 Tidelift Professional Open Source Survey https://cdn2.hubspot.net/hubfs/4008838/Introduction_to_Managed_Open_Source.pdf
57. TAKE A MINUTE TO INTERNALIZE THAT.
58. TAKE A MINUTE TO INTERNALIZE THAT. Synopsys: in 2017: 36% of code bases are open source components. in 2018: 57% of code bases are open source components.
59. TAKE A MINUTE TO INTERNALIZE THAT. Synopsys: in 2017: 36% of code bases are open source components. in 2018: 57% of code bases are open source components. Tidelift: in 2018: 60% of code bases are open source components, only 20% is custom application/business logic.
60. TAKE A MINUTE TO INTERNALIZE THAT. Synopsys: in 2017: 36% of code bases are open source components. in 2018: 57% of code bases are open source components. Tidelift: in 2018: 60% of code bases are open source components, only 20% is custom application/business logic.
61. Software now constructed from OSS puzzle pieces https://opbeat.com/blog/posts/picking-tech-for-your-startup/ Mike Krieger Instagram co-founder Borrow instead of building whenever possible There are hundreds of fantastic open-source projects that have been built through the hard experience of creating and scaling companies; especially around infrastructure and monitoring…that can save you time and let you focus on actually building out your product. Blog article: Advice on picking tech for your startup
62. Software now constructed from OSS puzzle pieces https://opbeat.com/blog/posts/picking-tech-for-your-startup/ Mike Krieger Instagram co-founder Borrow instead of building whenever possible There are hundreds of fantastic open-source projects that have been built through the hard experience of creating and scaling companies; especially around infrastructure and monitoring…that can save you time and let you focus on actually building out your product. Blog article: Advice on picking tech for your startup https://medium.com/@nayafia/open-source-was-worth-at- least-143m-of-instagram-s-1b- acquisition-808bb85e4681#.d6gzzr9nk
63. Yet, most have to self-support their OSS work OPEN SOURCE SURVEYS: 2018 Tidelift Professional Open Source Survey Over 1,200 respondents https://tidelift.com/about/2018-tidelift-professional-open-source-survey-results 62% of respondents said that they are required to financially support their open source work with their own funds, or that they receive no external funding at all.
64. A bit about open source NOW,
65. Of course, things could get terrifying from here
66. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE https://fordfoundcontent.blob.core.windows.net/media/2976/roads- and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf
67. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE https://fordfoundcontent.blob.core.windows.net/media/2976/roads- and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf Steve Marquess, noticed that one contributor, Stephen Henson, was working full time on OpenSSL. Curious, Marquess asked him what he did for income, and was shocked to learn that Henson made one-fifth of Marquess’s salary.
68. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE https://fordfoundcontent.blob.core.windows.net/media/2976/roads- and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf Steve Marquess, noticed that one contributor, Stephen Henson, was working full time on OpenSSL. Curious, Marquess asked him what he did for income, and was shocked to learn that Henson made one-fifth of Marquess’s salary. Marquess had always considered himself to be a strong programmer, but his skills paled in comparison to Henson’s. … Henson had been working on OpenSSL since 1998.
69. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE https://fordfoundcontent.blob.core.windows.net/media/2976/roads- and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf Steve Marquess, noticed that one contributor, Stephen Henson, was working full time on OpenSSL. Curious, Marquess asked him what he did for income, and was shocked to learn that Henson made one-fifth of Marquess’s salary. Marquess had always considered himself to be a strong programmer, but his skills paled in comparison to Henson’s. … Henson had been working on OpenSSL since 1998. “I had always assumed, (as had the rest of the world) that the OpenSSL team was large, active, and well resourced.” – Steve Marquess
70. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE https://fordfoundcontent.blob.core.windows.net/media/2976/roads- and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf Steve Marquess, noticed that one contributor, Stephen Henson, was working full time on OpenSSL. Curious, Marquess asked him what he did for income, and was shocked to learn that Henson made one-fifth of Marquess’s salary. Marquess had always considered himself to be a strong programmer, but his skills paled in comparison to Henson’s. … Henson had been working on OpenSSL since 1998. “I had always assumed, (as had the rest of the world) that the OpenSSL team was large, active, and well resourced.” – Steve Marquess In reality, OpenSSL wasn’t even able to support one person’s work.
71. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE https://fordfoundcontent.blob.core.windows.net/media/2976/roads- and-bridges-the-unseen-labor-behind-our-digital-infrastructure.pdf INDUSTRY, GOVERNMENT, ETC ARE OFTEN UNAWARE OF INFRASTRUCTURE’S FUNDING ISSUES Steve Marquess, noticed that one contributor, Stephen Henson, was working full time on OpenSSL. Curious, Marquess asked him what he did for income, and was shocked to learn that Henson made one-fifth of Marquess’s salary. Marquess had always considered himself to be a strong programmer, but his skills paled in comparison to Henson’s. … Henson had been working on OpenSSL since 1998. “I had always assumed, (as had the rest of the world) that the OpenSSL team was large, active, and well resourced.” – Steve Marquess In reality, OpenSSL wasn’t even able to support one person’s work.
72. In 2014, 66% of all web servers were using OpenSSL Meanwhile, OpenSSL was maintained by only a few volunteers HYPOTHETICAL WORST CASE http://veridicalsystems.com/ blog/of-money-responsibility-and-pride/
73. Truck Factor: the minimal # of developers that have to be hit by a truck (or quit) before a project is incapacitated ANOTHER WORST CASE Ever heard of the truck factor? https://peerj.com/preprints/1233.pdf - Look at the 133 most active projects on GitHub - Determine the amount of information concentrated in individual team members from commits.
74. Truck Factor: the minimal # of developers that have to be hit by a truck (or quit) before a project is incapacitated ANOTHER WORST CASE Ever heard of the truck factor? https://peerj.com/preprints/1233.pdf 64% OF PROJECTS RELIED ON 1-2 DEVS TO SURVIVE - Look at the 133 most active projects on GitHub - Determine the amount of information concentrated in individual team members from commits.
75. Truck Factor: the minimal # of developers that have to be hit by a truck (or quit) before a project is incapacitated ANOTHER WORST CASE Ever heard of the truck factor? Table 2: Truck Factor results TF Repositories 1 alexreisner/geocoder, atom/atom-shell, bjorn/tiled, bumptech/glide, celery/celery, celluloid/celluloid, dropwizard/dropwizard, dropwizard/metrics, erikhuda/thor, Eugeny/ajenti, getsen- try/sentry, github/android, gruntjs/grunt, janl/mustache.js, jr- burke/requirejs, justinfrench/formtastic, kivy/kivy, koush/ion, kriswallsmith/assetic, Leaflet/Leaflet, less/less.js, mailpile/Mailpile, mbostock/d3, mitchellh/vagrant, mitsuhiko/flask, mongoid/mongoid, nate-parrott/Flashlight, nicolasgramlich/AndEngine, paulas- muth/fnordmetric, phacility/phabricator, powerline/powerline, puphpet/puphpet, ratchetphp/Ratchet, ReactiveX/RxJava, sandstorm- io/capnproto, sass/sass, sebastianbergmann/phpunit, sferik/twitter, silexphp/Silex, sstephenson/sprockets, substack/node-browserify, thoughtbot/factory_girl, thoughtbot/paperclip, wp-cli/wp-cli 2 activeadmin/activeadmin, ajaxorg/ace, ansible/ansible, apache/cassandra, bup/bup, clojure/clojure, composer/composer, cucumber/cucumber, driftyco/ionic, drupal/drupal, elas- ticsearch/elasticsearch, elasticsearch/logstash, ex- cilys/androidannotations, facebook/osquery, facebook/presto, FriendsOfPHP/PHP-CS-Fixer, github/linguist, Itseez/opencv, jadejs/jade, jashkenas/backbone, JohnLangford/vowpal_wabbit, jquery/jquery-ui, libgdx/libgdx, meskyanichi/backup, netty/netty, omab/django-social-auth, openframeworks/openFrameworks, plataformatec/devise, prawnpdf/prawn, pydata/pandas, Re- spect/Validation, sampsyo/beets, SFTtech/openage, sparklemo- tion/nokogiri, strongloop/express, thinkaurelius/titan, ThinkU- pLLC/ThinkUp, thumbor/thumbor, xetorthio/jedis 3 bbatsov/rubocop, bitcoin/bitcoin, bundler/bundler, divio/django-cms, haml/haml, jnicklas/capybara, mozilla/pdf.js, rg3/youtube-dl, mrdoob/three.js, spring- projects/spring-framework, yiisoft/yii2 4 boto/boto, BVLC/caffe, codemirror/CodeMirror, gra- dle/gradle, ipython/ipython, jekyll/jekyll, jquery/jquery 5 iojs/io.js, meteor/meteor, ruby/ruby, WordPress/WordPress 6 chef/chef, cocos2d/cocos2d-x, diaspora/diaspora, em- berjs/ember.js, resque/resque, Shopify/active_merchant, spotify/luigi, TryGhost/Ghost 7 django/django, joomla/joomla-cms, scikit-learn/scikit-learn 9 JetBrains/intellij-community, puppetlabs/puppet, rails/rails 11 saltstack/salt, Seldaek/monolog, v8/v8 12 git/git, webscalesql/webscalesql-5.6 13 fog/fog 14 odoo/odoo 18 php/php-src 19 android/platform_frameworks_base, moment/moment 23 fzaninotto/Faker 56 caskroom/homebrew-cask 130 torvalds/linux 250 Homebrew/homebrew https://peerj.com/preprints/1233.pdf 64% OF PROJECTS RELIED ON 1-2 DEVS TO SURVIVE - Look at the 133 most active projects on GitHub - Determine the amount of information concentrated in individual team members from commits.
76. Truck Factor: the minimal # of developers that have to be hit by a truck (or quit) before a project is incapacitated ANOTHER WORST CASE Ever heard of the truck factor? Table 2: Truck Factor results TF Repositories 1 alexreisner/geocoder, atom/atom-shell, bjorn/tiled, bumptech/glide, celery/celery, celluloid/celluloid, dropwizard/dropwizard, dropwizard/metrics, erikhuda/thor, Eugeny/ajenti, getsen- try/sentry, github/android, gruntjs/grunt, janl/mustache.js, jr- burke/requirejs, justinfrench/formtastic, kivy/kivy, koush/ion, kriswallsmith/assetic, Leaflet/Leaflet, less/less.js, mailpile/Mailpile, mbostock/d3, mitchellh/vagrant, mitsuhiko/flask, mongoid/mongoid, nate-parrott/Flashlight, nicolasgramlich/AndEngine, paulas- muth/fnordmetric, phacility/phabricator, powerline/powerline, puphpet/puphpet, ratchetphp/Ratchet, ReactiveX/RxJava, sandstorm- io/capnproto, sass/sass, sebastianbergmann/phpunit, sferik/twitter, silexphp/Silex, sstephenson/sprockets, substack/node-browserify, thoughtbot/factory_girl, thoughtbot/paperclip, wp-cli/wp-cli 2 activeadmin/activeadmin, ajaxorg/ace, ansible/ansible, apache/cassandra, bup/bup, clojure/clojure, composer/composer, cucumber/cucumber, driftyco/ionic, drupal/drupal, elas- ticsearch/elasticsearch, elasticsearch/logstash, ex- cilys/androidannotations, facebook/osquery, facebook/presto, FriendsOfPHP/PHP-CS-Fixer, github/linguist, Itseez/opencv, jadejs/jade, jashkenas/backbone, JohnLangford/vowpal_wabbit, jquery/jquery-ui, libgdx/libgdx, meskyanichi/backup, netty/netty, omab/django-social-auth, openframeworks/openFrameworks, plataformatec/devise, prawnpdf/prawn, pydata/pandas, Re- spect/Validation, sampsyo/beets, SFTtech/openage, sparklemo- tion/nokogiri, strongloop/express, thinkaurelius/titan, ThinkU- pLLC/ThinkUp, thumbor/thumbor, xetorthio/jedis 3 bbatsov/rubocop, bitcoin/bitcoin, bundler/bundler, divio/django-cms, haml/haml, jnicklas/capybara, mozilla/pdf.js, rg3/youtube-dl, mrdoob/three.js, spring- projects/spring-framework, yiisoft/yii2 4 boto/boto, BVLC/caffe, codemirror/CodeMirror, gra- dle/gradle, ipython/ipython, jekyll/jekyll, jquery/jquery 5 iojs/io.js, meteor/meteor, ruby/ruby, WordPress/WordPress 6 chef/chef, cocos2d/cocos2d-x, diaspora/diaspora, em- berjs/ember.js, resque/resque, Shopify/active_merchant, spotify/luigi, TryGhost/Ghost 7 django/django, joomla/joomla-cms, scikit-learn/scikit-learn 9 JetBrains/intellij-community, puppetlabs/puppet, rails/rails 11 saltstack/salt, Seldaek/monolog, v8/v8 12 git/git, webscalesql/webscalesql-5.6 13 fog/fog 14 odoo/odoo 18 php/php-src 19 android/platform_frameworks_base, moment/moment 23 fzaninotto/Faker 56 caskroom/homebrew-cask 130 torvalds/linux 250 Homebrew/homebrew https://peerj.com/preprints/1233.pdf The higher the TF the better! Only a handful of projects with a high TF… TRUCK FACTOR RESULTS: 64% OF PROJECTS RELIED ON 1-2 DEVS TO SURVIVE - Look at the 133 most active projects on GitHub - Determine the amount of information concentrated in individual team members from commits.
77. A sampling of some low truck factors… TRUCK FACTOR RESULTS wp-cli/wp-cli sass/sass gruntjs/grunt mbostock/d3 ReactiveX/RxJava Truck Factor 2: apache/cassandra clojure/clojure netty/netty pydata/pandas drupal/drupal Truck Factor 1:
78. There exist high truck factors though too! ANOTHER WORST CASE
79. Ecosystem & Community is Everything LESSONS FROM
80. OOPSLA’13 Empirical Analysis of Programming Language Adoption Looked at lots of data. - 10 years of repository meta data, tracking up to 590,000 open source projects - Survey data of developers over multiple surveys, ranging from 1,000 to 13,000 respondents
81. OOPSLA’13 Empirical Analysis of Programming Language Adoption Looked at lots of data. - 10 years of repository meta data, tracking up to 590,000 open source projects - Survey data of developers over multiple surveys, ranging from 1,000 to 13,000 respondents Which factors most influence developer decision-making for language selection? BIG QUESTION:
82. Importance of different factors when picking a language: EMPIRICAL ANALYSIS OF PROGRAMMING LANGUAGE ADOPTION
83. OOPSLA’13 Empirical Analysis of Programming Language Adoption Which factors most influence developer decision-making for language selection? BIG QUESTION: Through multiple surveys, we saw that developers value open source libraries as the dominant factor in choosing programming languages. Social factors not tied to intrinsic language features, such as existing personal or team experience, also rate highly.
84. Empirical Analysis of Programming Language Adoption Which factors most influence developer decision-making for language selection? BIG QUESTION: Through multiple surveys, we saw that developers value open source libraries as the dominant factor in choosing programming languages. Social factors not tied to intrinsic language features, such as existing personal or team experience, also rate highly. (From the paper)
85. So you might say… A open source project ✨is✨ its Community.
86. So you might say… A open source project ✨is✨ its Community. ’s success attributed to
87. Professional users want an active community! OPEN SOURCE SURVEYS: 2018 Tidelift Professional Open Source Survey https://tidelift.com/about/2018-tidelift-professional-open-source-survey-results What do professional users care about most? - Software that is reliable and well maintained - Software that has an active community using and supporting it - Software that is secure - Software with maintainers who provide timely bug fixes and security releases
88. Professional users want an active community! OPEN SOURCE SURVEYS: 2018 Tidelift Professional Open Source Survey https://tidelift.com/about/2018-tidelift-professional-open-source-survey-results What do professional users care about most? - Software that is reliable and well maintained - Software that has an active community using and supporting it - Software that is secure - Software with maintainers who provide timely bug fixes and security releases Respondents rated an active community as being over 20% more important than the popularity of a project.
89. The shape of the ecosystem is important too! PAPER: Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects: A Case Study of the PyPI Ecosystem. Valiev, M., Vasilescu, B., and Herbsleb, J. ESEC/FSE 2018 Examined PyPI Ecosystem - Quantitative: analyzed 70,000 PyPI packages - Qualitative: interviewed 10 maintainers
90. The shape of the ecosystem is important too! PAPER: Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects: A Case Study of the PyPI Ecosystem. Valiev, M., Vasilescu, B., and Herbsleb, J. ESEC/FSE 2018 Examined PyPI Ecosystem - Quantitative: analyzed 70,000 PyPI packages - Qualitative: interviewed 10 maintainers
91. The shape of the ecosystem is important too! PAPER: Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects: A Case Study of the PyPI Ecosystem. Valiev, M., Vasilescu, B., and Herbsleb, J. ESEC/FSE 2018 Examined PyPI Ecosystem - Quantitative: analyzed 70,000 PyPI packages - Qualitative: interviewed 10 maintainers
92. What does this all mean? Community & Ecosystem are among the most important factors to an open source project’s success. TAKEAWAY: To stay alive, there should be a relentless focus on growing the community and ecosystem! And that is hard work.
93. Let’s wrap it up PHEW.
94. Things that are changing fast: How we build software Open Source Our idea of software engineers What we actually do nowadays when we sit down to build an app. The common infrastructure and tools that we all depend on What SWEs should know, how much experience they have, and who they are. & that more people should be aware of
95. Things that are changing fast: How we build software Open Source Our idea of software engineers What we actually do nowadays when we sit down to build an app. The common infrastructure and tools that we all depend on What SWEs should know, how much experience they have, and who they are. & that more people should be aware of We saw 3 areas…
96. What does this all mean? • We’re getting a lot of newcomers. • Those newcomers don’t all have fancy CS degrees. • But maybe they don’t need one? • Frameworks are king! • Applications now are only 20% business logic (in one estimate) • We’re getting a lot of newcomers. • We’re actually getting good at reuse nowadays. • The pieces that we (&newcomers) are reusing are largely open source • Corporate subsidy is helping open source a lot (49%) but 62% is self- funded/not funded • We’re getting a lot of newcomers.
97. It’s all related We’re getting and will keep getting a lot of newcomers across our industry Open source software continues to struggle with sustainability. Applications are becoming a majority open source components. Frameworks, and gluing together OS components.
98. It’s all related We’re getting and will keep getting a lot of newcomers across our industry Open source software continues to struggle with sustainability. Applications are becoming a majority open source components. Frameworks, and gluing together OS components. Newcomers depend on OSS, and on it being easy to use This is making it easier for anyone to build applications!
99. What does this all mean? As educators… As managers… As practicing engineers… Maybe everyone doesn’t need a CS degree. Maybe we have to figure out ways for people more quickly and affordably learn enough to become an application developer. Maybe this is OK. Your engineers are going to need to be good mentors. Put a greater emphasis on mentorship sooner than later. Find ways to allow your engineers to give back to the OSS you depend on. Mentorship is a thing you should drop everything and focus on getting better at.
100. Questions? THANK YOU!