Introduction

If you are part of the CFP group at PyCon India or any other conference, this post is for you. I have mentioned my learnings and working process for the CFP workgroup at PyCon India 2023.

I was one of the CFP (Call For Proposal) reviewers at PyCon (Python Conference) India 2019, Chennai. If you want to learn about my experience there, check out my article on The Inside Story of How We Selected 30 Talks from 280+ Proposals for Pycon India 2019.

After the 2019 edition, PyCon India will return to the offline conference in 2023, and Soumendra will also return as a volunteer. I have volunteered to be the CFP lead for the 2023 edition. We will briefly discuss the pre-event timeline.

Timeline

This is an overall timeline of the entire CFP process.
Before volunteering at PyCon, in my wildest dreams, I would have never thought that preparation needed to start 7-8 months before an event started.

Terms & Definitions

These terms will help you understand this post better.

Initial days 💡

• I joined as CFP Lead and core member for the PyCon India 2023 on 7 Apr 2023.

• I shared a CFP best practices written by previous organizers in 2019. I have also modified this guide down the lane based on the need. Future PyCon CFP reviewers can use this guide.

• On a Sunday evening, on 16 April, I organized the first CFP Reviewers meeting.

• Around 25 volunteers joined the call that evening.

Reviewer Management 🪄

As a CFP lead, you know the vital role of managing reviewers. This task is only second to managing proposals, which is the top priority. Your responsibilities include motivating volunteers to review, being empathetic, and ensuring work is completed throughout the five-month process. With your dedication and guidance, you can inspire your team to achieve great results.

Leading people may seem more manageable in a corporate setting, as I have previously managed teams of ten to fifteen individuals. However, managing volunteers who are not obligated to complete tasks and only work voluntarily without compensation presents a challenge.

Reviewers drop 😭

According to analysis, the reason why some reviewers left PyCon can be attributed to the following factors:

1. Some individuals were curious about how PyCon works and were not genuinely interested in contributing to the event. For all of the reviewers in this category, this blog post is for you.

2. As a Reviewer, one cannot submit a proposal to prevent any bias in the review process. This requirement caused many reviewers to leave.

3. The reviewer's job is both thankless and tiring, requiring evaluating each proposal for 30-45 minutes. To be a reviewer, one needs focus, dedication, and perseverance.

4. Volunteers for the conference do not receive free tickets or any additional facilities.

5. Other commitments may have made it difficult for some individuals to prioritize this volunteering activity.

6. It's possible that some reviewers didn't like the CFP lead. 🫣

Volunteer drop issue is common across Wikipedia, Mozilla, and other volunteer-driven projects I contribute to.

The trend of the number of attendees in the weekly call of the CFP workgroup.

75+ volunteers signed up to be reviewers. Twenty-five joined on the first call, and by the end of five months, there were around six left. With a lower Reviewer: Proposal ratio, later on, we moved to generalists rather than experts to review the proposals.

One positive aspect of volunteer-led initiatives is that the individuals who remain committed until the end are usually highly motivated. They can often surpass expectations by ten to one hundred times. If their opinions are respected and granted autonomy, they will likely exceed expectations even further.

Proposals Management 📒

To know more about the proposal category trends across previous editions, please follow my blog post, PyCon India 2023 Proposals Analysis (soumendrak.com).

Once the Junction tool was ready with data backup of previous editions and Category <> Reviewer mapping, we opened the CFP submission.

The trend of the number of Talk and Workshop proposals received over time.

There is a spike of proposals towards the first deadline, i.e., 5 Aug. Then a slight spike in the final deadline, i.e., 15 Aug.

Initially, I was a bit unhappy that with only three days left, we got around 150 proposals, which is half of the proposals we received in 2019. However, due to aggressive promotion from all the social media handles and local chapters, we received 23 submissions on 4 Aug and 53 proposals 🤯 on 5 Aug, which was quite overwhelming. However, we could not break the number of proposals milestones in 2019.

Proposals race 🐎

We have received a total of 303 proposals with the following distribution:

• 7 Devsprints proposals

• 11 Birds of Features proposals

• 14 Lightning Talk proposals

• 42 Workshop proposals

• 229 Talk proposals

Let's see the qualifying rate.

• 35 Talk proposals got shortlisted. This suggests that every proposal has only a 15% (35/229) chance of being shortlisted.

• For Workshop proposals, it was 17%. Out of 42, 7 were selected.

• For Lightning talks, it was 42%.

• For Devsprint, 100% of all proposals were selected.

Proposal review process

Let's break down how the Talk proposals narrowed down over the rounds.

Number of Talk proposals qualified across various stages of the review process

1st round

• We checked the minimum resources on 1st round. Whether the proposals have enough resources to make a fair judgment, which is 3S's:

  • Slides (if any)

  • Source code, and

  • Speaker profile.

• We have allowed the submission of proposals till the end of August. There were medical cases where we made exceptions.

• In 2019, we received over 280 proposals in this process stage and unfortunately had to reject around 150. We aimed to decrease the number of rejections to below 50 for this round in this edition. To ensure proposers knew their status, we sent multiple reminders through Junction. These reminders were sent one week, two weeks, and one month after the initial submission.

• However, we did receive feedback from some proposers who were not receiving emails about Junction's comments. As a result, we sent one final reminder email to all proposers to double-check their review comments.

• Unfortunately, 99 of the proposals were not able to undergo review due to insufficient materials provided.

2nd round

• We have thoroughly reviewed the remaining 130 proposals a minimum of three times. Using the Junction tools, we ensured that at least three reviewers voted for every proposal.

• Each proposal was scored from -2 (lowest) to 3 (highest). However, due to time constraints and the absence of a minimum of three reviewers in all categories, we sought additional assistance and proceeded to the third round.

3rd round

• We needed additional reviewers for this round, so the Core team members and experienced volunteers who had previously done reviews joined the CFP team to assist. We reviewed proposals missed in Junction, specifically those with less than three reviewer votes.

• We also reviewed any proposals marked as Promotional. Our goal was to ensure that at least three reviewers reviewed all second-round qualified proposals by the end of this round.

• Once all proposals were scored, we proceeded to sort and further process those that were selected for the fourth round.

4th round

• Based on the distribution of the proposals across different categories, a guideline has been prepared to decide how many proposals we should select across each category.

  $$Ntsx = Nts * \frac{Nrx}{Nt}$$

  Where,
  Ntsx = Number of proposals to be selected from category x
  Nts = No of total Talks to be chosen for the conference
  Nrx = Number of proposals received in category x
  Nt = Total number of proposals received

• This was used as a guideline. All proposals were selected based on their merit. Suppose for a category, the formula was showing 0; if the proposal was highly scored, we chose that adjusting from other categories.

• Diversity and First-time speaker preference came into the picture here. Extra preference was given to the Talks with first-time speakers or Female speakers.

• We prepared for the final round.

5th and final round

• Later, we learned that seven more proposals could be selected. We added the seven high-scored proposals to the list.

• Two of the speakers from Hyderabad who are coming to the conference have been shortlisted as standby speakers.

• For one last time, we have gone through the final shortlisted Talks.

A similar approach was followed for Workshop proposals.

The challenge was to complete the first two rounds in parallel. While the proposals were being prepared for the second round, we assigned experts in that proposal category as reviewers.

Post Talks selection

Speaker rehearsals

• Once the speakers accepted our invitation, the speaker rehearsal process began. However, due to my academic deadlines, I couldn't focus on it much. I want to thank Samir and Koti for stepping up and filling the gap.

• In this process, we listened to the speakers' short rehearsal and provided suggestions for improvements.

Scheduling the Talks

• Scheduling the Talks is always a challenge. In addition, we learned that the tracks will be in different buildings, with a good distance between them.

• This was my first time as a scheduler; I was listening to the challenges involved most of the time. I want to thank Praneet Mokkapati, who helped us identify future issues and schedule the Talks.

• The sole aim was to reduce the movement between tracks and let attendees get time to interact with one another.

Tools and Technologies ⚒️

This Section is for you if you are a future PyCon India or any other conference organizer. The following tools helped make the conference possible.

1. Junction: A Django-based web application for proposals and reviewer management.

2. Jitsi: A free web and mobile app to start and join meetings.

3. Zoho Mail is a free mailbox for sending and receiving emails with a custom domain like cfp@in.pycon.org, tickets@in.pycon.org, or contact@in.pycon.org.

4. GitHub Project Manager: We have used the GitHub project manager to manage the workgroup tasks.

5. Google Docs: We have extensively used Google Docs for Agenda, MoM, and other free-flowing text-related documents.

6. Google Sheets: We have used Google Sheets extensively from day 1 to manage the proposals, categorize and classify them, etc.

7. Google Forms: We have used Google Forms to collect reviewer, proposer, and speaker input at various conference phases.

8. Netlify: We have used Netlify for PR review and our blog and website deployment.

9. NextJS: We have used Next.js for the main website.

10. Calendly: For speaker rehearsal slot booking.

Lessons Learnt 📋

In these seven months, I learned many things:

1. Self-realization: I will do it if I have committed to delivering task A, which significantly impacts other human beings. It does not matter if I am getting paid for it or not, or if I get support from others.

2. Asynchronous communication: For the CFP workgroup, I tried to manage most of the tasks asynchronously over Zulip and WhatsApp rather than frequent meetings where all participants needed to meet. During the later parts, I made weekly calls and took asynchronous updates from reviewers. Too many synchronous meetings (every participant needs to be there simultaneously) are for people with poor written communication skills, lack of motivation, or less accountability.

3. Better planning of tasks: I should have divided the proposals and assigned them to individuals to review rather than asking them to pick up from their preferred category. Often, people want to get specific tasks, and someone else takes care of the bigger picture.

4. Time and Energy Management: Time and Energy management across office work, M.Tech, my other volunteering work, and PyCon was challenging. Active prioritization and energy management helped me throughout the journey, and this also helped during the event days.

5. Proposer Threats: Some proposers did not like that their proposal was not selected. One even threatened to complain against me in the Indian ministry. I ignored the threat. Through years of experience in social media and online forums, I have learned where to give my attention.

6. Junction Tool: The Junction tool is not fully matured. After a particular stage in the review process, we were forced to migrate to Google Sheets for proposals and reviewer management. As the only consumer with superuser access in Junction, I will help the Junction lead, Ananyo, to add more features.

7. Attendees' preference: While buying tickets, we could have asked the attendees about the top 3 categories of Talks they wanted to attend, but we missed adding that. That would have made our job easier to select proposals across categories.

8. Pressure on last rounds: The previous three rounds were conducted in seven to ten days. I should have planned better and reduced the number of Talks needed to qualify for further rounds.

9. Attention to other tracks: Other than talks and workshops, I did not pay any attention to poster presentations, software exhibits, or promotion of other tracks; this could have been improved.

10. Proposals and Generative AI: A few proposals have been made entirely out of chatGPT, and even the source code went undetected till the later stage of the conference. For future CFP teams, it will be trickier to identify whether AI made an excellent first impression but, deep down, low-quality proposals.

11. Making everyone happy: One thing that became clear was that we would not be able to make every attendee pleased with the proposal selection or schedule preparation. However, we have tried our best to give the majority of attendees an excellent, unforgettable experience.

Acknowledgment

Volunteers at PyCon India 2023 closing ceremony. P.C.:Animesh Ghosh

PyCon India is run entirely by volunteers who bring in sponsors, organize the event, and even pay for their tickets. As a core team member, I have witnessed the remarkable effort and energy a few dedicated volunteers put into the conference. This demonstrates the strength of human beings, as not all actions are driven by monetary gain. When a cause is inspiring enough, it can unite many individuals to work towards a common goal, whether the Indian independence movement or organizing an event. The reason is what truly matters. I want to thank the other workgroup leads and volunteers for doing their responsibilities to make this conference possible.

I am thrilled to extend my heartfelt appreciation to my fantastic co-lead, Dr. Murali, who stepped up and seamlessly took on my duties whenever urgent matters cropped up over seven months.

🎖️Thanks to the heart and soul of the workgroup, Aman Pandey, Ashok Tankala, Koti Vellanki, Mohit Bansal, Pavan G, and Samir Thakkar, without whom it would not have been possible to review 300+ proposals.

I thank the platform lead, Ananyo, who had been resolving all the Junction issues within 24 hours to unblock the CFP team.

✨ Special thanks to Bhanu, Chandan, and Kalyan, who stood up when needed.

Finally, I would like to thank the conference chair, Sanchit Balchandani, for allowing me to lead the CFP workgroup.

Conclusion

I volunteered for this edition of PyCon for

a) PyCon India is back to the offline conference after a gap of 4 years,
b) I wouldn't say I liked that PyCon India did not happen last year in 2022 for the first time since its inception in 2009.

On the days of the event, I contributed to the registration desk session monitor/chair at various tracks. Maybe I will share those experiences in another post.

In this post, we discussed the overall process of the CFP workgroup since its inception and the challenges faced in between. Thank you.

Partial CFP Team: (From left) Samir Thakkar, Koti Vellanki, Soumendra (me) and Dr Murali

A few other resources like Email templates and plans: PyCon India 2023 Dashboard (soumendrak.com)