Wednesday, March 20, 2024

Announcing a PyPI Support Specialist

We launched the Python Package Index (PyPI) in 2003 and for most of its history a robust and dedicated volunteer community kept it running. Eventually, we put a bit of PSF staff time into the maintenance of the Index, and last year with support from AWS we hired Mike Fiedler to work full-time on PyPI’s urgent security needs.

PyPI has grown enormously in the last 20+ years, and in recent years it has reached a truly massive scale with growth only continuing upward. In 2022 alone, PyPI saw a 57% growth and as of this writing, there are over a half a million packages on PyPI. The impact PyPI has these days is pretty breathtaking. Running a free public service of that size has come with challenges, too. As PyPI has grown, the work of communicating with users and solving account issues here has grown in tandem and out-stripped our current volunteer plus one tenth of a staff person capacity. We also know that some community members have noticed and expressed frustration with the time-frame that goes with tasks that don't have sufficient staffing.

Much of this work is sensitive and complex such that it needs to be performed by a PSF staff person. It involves personal information and verification processes to make sure we’re giving access and names to the correct entities. Work like this needs to be done by a person who is here day after day to carry out multi-step verification procedures and is accountable to the PSF. 

We are very happy to share the news that we are hiring a person to help us manage the increased capacity and allow us to keep pace with PyPI’s seemingly unstoppable growth. This is an associate role that is 100% remote. Please take a look at this posting for a PyPI Support Specialist and share it with your networks.

Thursday, February 29, 2024

White House recommends use of memory-safe languages like Python

Earlier this week the White House published a report recommending the use of memory-safe programming languages to eliminate an entire class of vulnerabilities affecting software. The report quotes claims from large software producers like Google and Microsoft which estimate that 70% of vulnerabilities affecting software are due to memory-safety issues.

Back in December of 2023, the Cybersecurity and Infrastructure Security Agency (CISA) published a report that included a list of memory-safe programming languages, among them was the Python programming language.

The Python Software Foundation’s response to the US Government's Request for Information noted Python's memory-safety and ability to wrap code written in C, C++, and Rust among other systems languages. Part of Python’s popularity stems from the large number of community-maintained packages using this feature for performance, wrapping existing libraries, and low-level API access.

Cryptography is one of the most depended on Python libraries for cryptographic primitives, installed nearly 10 million times per day. Cryptography started migrating from using C to Rust for security reasons in 2020 and made the first release with Rust binary extensions in 2021. You can listen to maintainers Paul Kehrer and Alex Gaynor discuss this non-trivial migration in their PyCon 2022 talk.

The migration of the cryptography library included tools like PyO3 and setuptools-rust that enable easier adoption of Rust binary extensions. There’s already plenty of buzz for using Rust and Python together, the adoption of Rust in Python packages is steadily increasing from the single digits in 2020 to today with hundreds of packages using Rust.

There are many opportunities to learn about writing Python binary extensions using Rust, for example, at PyCon US 2024 there will be a tutorial about getting started with PyO3 and a talk on PyO3 and maturin, a PEP 517 build backend for Rust by a maintainer of the PyO3 project.

Historically Python binary extensions were built mostly using C and C++ meaning there are many projects which, for reasons like backwards compatibility or lack of resources and time, cannot or do not want to migrate to Rust. For these projects, the use of compiler options can harden binaries against some memory safety issues. The OpenSSF Best Practices working group has published a list of compiler options to consider adopting in order to harden builds of C and C++ code.

There is still much work to be done to secure the Python ecosystem and it can’t be done without our amazing community of contributors and maintainers. We look forward to more investment in this area as part of the industries’ adoption of memory-safe programming languages. If you are interested in being part of conversations around improving security in Python, we invite you to open a thread on discuss.python.org.