Reconstructing Taiwan's Internet-Based Party Politics by Ruby

In an era where most activities in life can be executed online, the majority of countries still lack the ability to interact politically via the internet, such as online voting, party membership registration, and petition signing.

In Taiwan, a political party named “New Power Party” has, since its establishment in 2015, relied on volunteers to develop a complete system enabling online party membership, donations, and voting. After several years of developing and operating in a dynamic and evolving manner, the system accumulated substantial technical debt. This led to issues with proper and complete system functionality and maintenance, impacting the party’s operations. Eventually, we decided to re-evaluate the specifications and rebuild the entire system using the latest versions of Ruby and Rails.

In this session, I will explore the process of revamping an outdated Rails application, lacking in testing and specification documentation, with the new Ruby 3 and Rails 7. This will include code refactoring, selecting the Runtime Stack, and integrating automated deployment processes using Ruby-based tools. I will also share practical experiences and reflections on integrating data from old and new versions and relaunching the system.

Transcript

1. Mu-Fan @ RubyConf AU 2024 Reconstructing Taiwan's Internet-Based Party Politics
3. Self Introduction • Mu-Fan Teng • Ruby Evangelist in Taiwan. • Organize most Ruby Community Events in Taiwan. • CEO of 5xruby.com • Twitter / GitHub: @ryudoawaru
5. RubyConf Taiwan • 2010~ • One of Matz's "Major" Ruby Conference choices outside Japan • Featured Matz all the time after 2012
7. Regular Ruby Meetups in Taiwan • Monthly Event @ PicCollage O ffi ce, Taipei. • Sharing and networking. • If you happen to be in Taiwan, please remember to join. • @rubytaiwan
8. COSCUP 2024 Ruby Track 2024/8/3~4, Taipei • Community-led agenda track at COSCUP. • Scheduled for August 3-4 at National Taiwan University of Science and Technology, Taipei. • Call for proposals now open. • CFP open now at: https:// pretalx.coscup.org/ coscup-2024/
9. AGENDA • Introduction of New Power Party and major political parties' internet participation in Taiwan. • Development history and issues of the legacy version. • Challenges and problems encountered when creating the new version.
10. New Power Party（NPP, 時代 力 量） • Was one of the major political parties in Taiwan. • Known for its internet participation. • Maximum number of active party members was around 2,000.
11. Main Features of NPP Website Membership Application Membership Due Payment Donation Internal Election CMS
12. Party Joining Process • Fill in basic information. • Pay online or O ff l ine in local branches. • Become a member after Review & Approve by Headquarter Sta ff s.
13. Membership Expiration Rules • Annual membership is similar to a subscription service. • Divided by fi rst and second half of the year. • Unpaid memberships will enter an expired state. • Membership can be restored by paying within two years. • After two years, it will change to a withdrawa state.
14. Donations Diverse Ways to Support the Party • One-time • Premium • Monthly Recurring
15. NPP’s Distinctive Approach Fully Online Elections for Core Decision-Making Bodies • Decision Making Committee (DMC) • Serves as the primary operational decision-making body. • Elects the party chairperson from among its members. • Party Representative Committee (PRC) • A statutory body required by Taiwanese party law. • Convenes semi-annually for tasks like amending the party constitution and ratifying fi nancial reports.
16. Party Election Process
17. Legacy Version Development Status • Built with Ruby on Rails from day 1. • First version completed by volunteers. • Later taken over by a full-time Junior Engineer for development. • Took over by 2021 after the last engineer quits.
18. Issues of Legacy Version—Project Management • No professional project management from the start. • Many features developed but rarely or never used. • Senior members unclear on the purpose of numerous functions. • Over-engineered features with unnecessary complexity and poor system design.
19. Issues of Legacy Version—Data Integrity • Frequent errors due to data errors. • DB Migration cannot be executed normally.
20. The History of Remaking Decision Making • Took over project maintenance in 2021, focusing on front-end and infrastructure. • End of 2021, faced challenges due to party charter amendments and dues calculation changes. • Spent considerable time on remediation and data adjustments. • Initiated project rebuild in Q3 of 2022, o ffi cially starting in October 2022.
21. Application Stack Changes Legacy New Ruby 2.6 3.1 Rails 5.2 7.0 Database PostgreSQL 9 PostgreSQL 14 Worker Sidekiq 5 + Redis GoodJob CSS Framework Bootstrap 3 TailwindCSS 3 + CSS Bundling JS Framework jQuery Stimulus JS + Importmap Deploy Tool Capistrano Docker Swarm
22. Main Improvement Goals for the New Version • Normalize the database schema. • Retain only essential functions. • Isolate election functionality, enhancing security and reliability. • Achieve >80% unit test coverage, including feature tests for critical functions. • Improve DevOps practices with CI/CD automation. • Integrate data from the legacy system into the new system.
23. Schema Improvement — User, Membership and Due
24. Use AASM to Reduce Code Coupling and Complexity
25. Before class User def set_status if ["friend", "member_unchecked", "member_unchecked_paperworked", "member"].include? (self.classi fi cation.to_s) if con fi rmed_membership_dues.any?(&:lifelong?) && !self.vip? && self.member? self.vip! self.issue_membership_number if self.membership_number.blank? update_columns membership_expires_on: LIFELONG_MEMBERSHIP_EXPIRES_ON return self.membership_number end last_due = con fi rmed_membership_dues.last # …… end end
26. After class Membership aasm column: :state do event(:pay) do before { |due| extend_by_due(due) } after_commit { |due| send_applied_mail(due) } transitions from: :applying_unpaid, to: :applying_paid transitions from: :legal, to: :legal transitions from: :expired, to: :legal end event(:cancel_applying, after_commit: :send_cancel_mailer) do transitions from: %i[applying_unpaid applying_paid], to: :applying_cancelled end end end class MembershipDue aasm column: :state do state :pending, initial: true state :paid event :pay, before: :before_pay, after: :after_pay, guard: :payable? do transitions(from: :pending, to: :paid) end end end
27. Improvement Results: Lines of Code Model Before After Di f User 1,212 190 -1,022 Membership 0 225 225 MembershipDue 369 157 -212 Total 1,581 572 -1,009 ＊W/O Comment, Concerns Included.
28. Issues with the Legacy Election Functions • Need to manually generate voter lists, noti fi cations, and ballots using rake tasks • Voting can be done just by accessing the ballot URL, although part of the ballot URL uses random numbers and alphabets, the identity of the voter cannot be ensured. • Ballot data is not encrypted, so risks cannot be ensured in case of data leakage.
29. Election System Refinement Plan • Separate the election functionality as an independent website. • Ensure compatibility with both new and old website logins and data. • Complete the development within one month due to unexpected political schedule changes.
30. OAuth Sign In • Use Doorkeeper Gem to establish OAuth Authorization Server on both legacy & new sites. • Respond with the same OAuth Raw Info through API.
31. Member Data Integration • Synchronize member data from the o ffi cial website for voter roll creation. • Initially used database syncing from the legacy version to the election site’s Voter Model due to tight development schedules. • Adopt RESTful integration with the new o ffi cial website for subsequent updates.
32. Encrypting Ballot Data • The ballot data stores the voter's member number. • Use ActiveRecord Encryption to encrypt the ballot data to prevent leakage of voting details.
33. Automate All Batch Processing • Write ActiveJob to do the batch tasks. • Use ActiveJob's set_wait to schedule corresponding Jobs based on the creation time of the election data.
34. The Test Coverage • Developed by 2 members in 1 month. • Achieved 81% test coverage.
35. DevOps Improvement
36. Concrete Goals • When submitting a Pull Request: • Automatically scan and check the code for syntax and security issues. • Automated testing. • Automatically generate an executable environment for User Testing (Review Apps). • When a Pull Request is merged into the main branch: • Can automatically deploy to staging environment. • Can semi-automatically deploy the program to the production through CI/CD Job.
37. Actions Needed to Achieve Goals • Dockerize the project. • Write CI/CD pipeline jobs to automate the work fl ow. • Choose an appropriate runtime stack.
38. Gems For Dockerize The Project Quickly • boxing：Generate a Docker fi le for the project. • openbox：Generates the Entrypoint for Docker. • liveness：Generates a health check endpoint.
39. boxing Gem • https://rubygems.org/gems/boxing • Multi-stage builds. • Use Alpine Linux to compact the image size. • DSL & Ruby way to set up Docker fi le. # con fi g/boxing.rb Boxing.con fi g do |c| c.revision = true c.sentry_release = true c.health_check = true c.build_packages = %w[gcompat] end
40. openbox Gem • https://github.com/elct9620/openbox • Checks DB connection automatically before starting any command • Easy to extend new commands • Use the same entrypoint with di ff erent entrypoint commands # Docker fi le ENTRYPOINT ["bin/openbox"] # docker-compose.yml version: '3.8' x-application: &application image: "${IMAGE_NAME}:$ {IMAGE_TAG}" #... services: # Default entrypoint: 'bin/openbox server' application: <<: *application good_job: <<: *application command: good_job
41. liveness Gem • The Rack middleware to provide health check endpoints • Con fi gurable by DSL • Easy to extend check provider and add endpoint security like token and IP whitelist # con fi g/routes.rb Rails.application.routes.draw do mount Liveness::Status => '/status' end # con fi g/initializers/liveness.rb Liveness.con fi g do |c| c.add :postgres, timeout: 10 c.add :redis, timeout: 10 end
42. Docker Compose File version: '3.8' x-application: &application image: "${IMAGE_NAME}:${IMAGE_TAG}" environment: - AUTO_MIGRATION=yes services: good_job: <<: *application deploy: replicas: ${WORKER_REPLICAS:-1} command: good_job healthcheck: test: ["NONE"] application: <<: *application deploy: replicas: ${APPLICATION_REPLICAS:-1} labels: - trae fi k.enable=true - trae fi k.http.routers.${DEPLOY_NAME}-https.rule=Host(`${DEPLOY_DOMAIN}`) - trae fi k.http.routers.${DEPLOY_NAME}-https.entrypoints=https - trae fi k.http.services.${DEPLOY_NAME}.loadbalancer.server.port=3000 networks: - trae fi k-public
43. Review Apps Provide an automatic live preview of changes made in a feature branch
44. Predefined CI/CD variables review:deploy: when: manual extends: .deploy stage: deploy environment: name: review/$CI_COMMIT_REF_NAME #prede fi ned variable on_stop: review:stop # Job name for stopping the app variables: # $CI_COMMIT_REF_NAME is one of prede fi ned vars DEPLOY_NAME: npp2023-$CI_ENVIRONMENT_SLUG tags: # Ensure Run on Docker Swarm - swarm-deploy only: - merge_requests needs: - docker - job: assets:precompile artifacts: true
45. Use Templates to Templatize Repeated Configurations • https://github.com/elct9620/ruby-gitlab-ci • GitLab CI templates for Ruby and Rails projects. • Contains many examples for reference. • Supports deployment via Trae fi k and Docker Swarm Mode.
46. What it Looks Like（Pull Request）
47. What it Looks Like（PR Merged）
48. Data Migration
49. Encapsulate Migration Code as a Rails Engine • The Migrator works for: 1. Migrate Database. 2. Migrate Carrierwave Attachment to Active Storage. • Developed as an independent Rails Engine Gem. gem 'legacy', git: 'https://gitlab.com/newpowerparty/legacy_migrator.git', branch: 'main'
50. Migrator Implementation: Base Model class Legacy::ApplicationRecord < ActiveRecord::Base self.abstract_class = true establish_connection ENV['LEGACY_DATABASE_URL'] scope :scope2export, -> { all } def skip?; false; end # De fi ne conditions if do want this row to be converted def migrate; raise NoMethodError; end def migrate! migrate.save!(validate: self.class.const_get(:VALIDATE_ON_MIGRATING), callbacks: self.class.const_get(:SAVE_WITH_CALLBACKS)) end class << self def after_migration; return; end def migrate! ::ApplicationRecord.transaction do scope2export. fi nd_each do |lrec| unless lrec.skip? new_record = lrec.migrate! lrec.record_migration!(new_record) end end rescue StandardError => e logger.info e.inspect end after_migration
51. Migrator Implementation: Attachments class Legacy::FileVault < ApplicationRecord mount_uploader : fi le, GeneralUploader scope :scope2export, -> { preload(:creator, :updator).all } def migrate @new_ fi le_vault = ::FileVault.new(id:, legacy_creator_name: creator&.name, legacy_updater_name: updator&.name) @new_ fi le_vault. fi le.attach(io: fi le.sanitized_ fi le. fi le, content_type: fi le.content_type, fi lename: fi le. fi le. fi lename ) @new_ fi le_vault end def skip? begin fi le.cache_stored_ fi le! # Skip the record if the fi le does not exist rescue Errno::ENOENT return true end false end
52. Use Git Token When Building Image # Docker fi le ARG APP_ROOT=/src/app ARG RUBY_VERSION=3.1.3 FROM ruby:${RUBY_VERSION}-alpine AS gem ARG APP_ROOT ARG GITLAB_TOKEN_NAME ARG GITLAB_TOKEN_TOKEN ENV BUNDLE_GITLAB__COM=$ {GITLAB_TOKEN_NAME}:${GITLAB_TOKEN_TOKEN} ...
53. On Launch Day • Migration took about 30 minutes • Including subsequent data veri fi cation, it took about 2 hours • Re-entering news and other CMS data took about 2 days • No data loss issues occurred
54. THANK YOU