Performance problems hide in plain sight. A controller action taking 800ms feels "fast enough" during development testing single requests, yet becomes unacceptable when fifty concurrent users generate hundreds of requests per minute. Database queries running in 150ms barely register during development with small datasets, yet balloon to seconds when production databases grow to thousands of records. Without visibility into actual performance, degradation sneaks in gradually until users complain.
Week 46 implemented comprehensive performance monitoring infrastructure providing automatic detection of slow queries and requests without expensive third-party services. The system logs slow queries with stack traces identifying problematic code, times every request measuring controller action duration, sends email alerts when configurable thresholds exceed limits, and provides rake tasks analyzing logs to suggest missing database indexes. This article explores the monitoring implementation, email alert system, profiling tools integration, and database performance analysis.
What Your Team Will Notice
Development logs now show clear performance warnings whenever queries exceed 100ms or requests exceed 500ms (configurable thresholds). Slow query warnings include full stack traces showing exactly which model method or controller action triggered the problem, the SQL query itself for analysis, and execution time highlighting how slow the query ran.
Email alerts arrive immediately when performance problems occur in production. Slow request alerts show the controller action, execution time, request parameters (sanitized), and timestamp. Slow query alerts include the SQL statement, execution time, stack trace showing calling code, and affected table information. These alerts enable rapid response—developers know about performance problems within seconds rather than waiting for user reports.
New rake tasks provide database optimization guidance: rake performance:check_indexes analyzes models suggesting which indexes might improve query performance based on common query patterns, and rake performance:analyze_queries processes Rails logs identifying the slowest queries and suggesting optimization approaches. These tasks turn abstract performance monitoring into actionable optimization tasks.
Development profiling tools integrate seamlessly: flamegraph and stackprof gems enable detailed CPU profiling when investigating specific performance problems, generating visual flamegraph SVGs showing where code spends time, and providing detailed profiling reports for optimization work. These tools sit dormant until needed, available instantly when investigation requires them.
The monitoring infrastructure operates transparently—no code changes required in controllers or models, no manual instrumentation scattered through business logic, and no performance overhead when everything runs quickly. Monitoring activates only when problems occur, making itself invisible during normal operation whilst providing crucial visibility during performance degradation.
Under the Bonnet: Request Timing with Around Action
Controller request timing uses around_action callbacks:
# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
around_action :log_performance
private
def log_performance
start_time = Time.current
yield
ensure
duration = ((Time.current - start_time) * 1000).round(2)
if duration > slow_request_threshold
log_slow_request(duration)
send_slow_request_alert(duration) if Rails.env.production?
end
end
def slow_request_threshold
ENV.fetch('SLOW_REQUEST_THRESHOLD', 500).to_i
end
def log_slow_request(duration)
Rails.logger.warn(
"[SLOW REQUEST] #{controller_name}##{action_name} " \
"took #{duration}ms " \
"(threshold: #{slow_request_threshold}ms)"
)
Rails.logger.warn(
" Parameters: #{sanitize_params(params.to_unsafe_h)}"
)
end
def send_slow_request_alert(duration)
PerformanceMailer.slow_request_alert(
controller: controller_name,
action: action_name,
duration: duration,
params: sanitize_params(params.to_unsafe_h),
timestamp: Time.current
).deliver_later
end
def sanitize_params(params_hash)
# Remove sensitive parameters
params_hash.except('password', 'password_confirmation', 'token', 'api_key')
end
end
This callback measures every request transparently, logging slow requests and sending alerts when thresholds exceed, without polluting controller code with timing logic.
Automatic Slow Query Detection
Database query monitoring patches ActiveRecord:
# config/initializers/performance_monitoring.rb
module QueryPerformanceMonitoring
def exec_query(sql, name = nil, binds = [], prepare: false)
start_time = Time.current
result = super
ensure
duration = ((Time.current - start_time) * 1000).round(2)
if duration > slow_query_threshold
log_slow_query(sql, duration, caller)
send_slow_query_alert(sql, duration, caller) if Rails.env.production?
end
result
end
private
def slow_query_threshold
ENV.fetch('SLOW_QUERY_THRESHOLD', 100).to_i
end
def log_slow_query(sql, duration, call_stack)
Rails.logger.warn(
"[SLOW QUERY] #{duration}ms (threshold: #{slow_query_threshold}ms)"
)
Rails.logger.warn(" SQL: #{sql}")
Rails.logger.warn(" Called from:")
# Show relevant stack trace (skip framework internals)
relevant_stack = call_stack
.reject { |line| line.include?('activerecord') }
.reject { |line| line.include?('activesupport') }
.first(5)
relevant_stack.each do |line|
Rails.logger.warn(" #{line}")
end
end
def send_slow_query_alert(sql, duration, call_stack)
EmailNotifier.slow_query_alert(
sql: sql,
duration: duration,
stack_trace: call_stack.first(10),
threshold: slow_query_threshold,
timestamp: Time.current
)
end
end
# Patch ActiveRecord connection adapters
ActiveRecord::ConnectionAdapters::AbstractAdapter.prepend(QueryPerformanceMonitoring)
This monkey patch intercepts all database queries, timing them automatically and alerting when slow queries occur without requiring explicit instrumentation.
Email Alert System
The email notifier handles alert delivery:
# app/services/email_notifier.rb
class EmailNotifier
def self.slow_query_alert(sql:, duration:, stack_trace:, threshold:, timestamp:)
return unless should_send_alert?
PerformanceMailer.slow_query_alert(
sql: sql,
duration: duration,
stack_trace: stack_trace,
threshold: threshold,
timestamp: timestamp
).deliver_later
end
def self.slow_request_alert(controller:, action:, duration:, params:, timestamp:)
return unless should_send_alert?
PerformanceMailer.slow_request_alert(
controller: controller,
action: action,
duration: duration,
params: params,
timestamp: timestamp
).deliver_later
end
private
def self.should_send_alert?
Rails.env.production? &&
ENV['PERFORMANCE_ALERT_EMAIL'].present? &&
ENV['PERFORMANCE_ALERT_FROM_EMAIL'].present?
end
end
# app/mailers/performance_mailer.rb
class PerformanceMailer < ApplicationMailer
def slow_query_alert(sql:, duration:, stack_trace:, threshold:, timestamp:)
@sql = sql
@duration = duration
@stack_trace = stack_trace
@threshold = threshold
@timestamp = timestamp
@environment = Rails.env
mail(
to: ENV['PERFORMANCE_ALERT_EMAIL'],
from: ENV['PERFORMANCE_ALERT_FROM_EMAIL'],
subject: "[#{@environment.upcase}] Slow Query Alert: #{@duration}ms"
)
end
def slow_request_alert(controller:, action:, duration:, params:, timestamp:)
@controller = controller
@action = action
@duration = duration
@params = params
@timestamp = timestamp
@environment = Rails.env
mail(
to: ENV['PERFORMANCE_ALERT_EMAIL'],
from: ENV['PERFORMANCE_ALERT_FROM_EMAIL'],
subject: "[#{@environment.upcase}] Slow Request Alert: #{@controller}##{@action} (#{@duration}ms)"
)
end
end
This service centralizes alert logic, ensuring consistent alert behaviour whilst respecting environment configuration.
Email Templates
Alert emails provide actionable information:
<!-- app/views/performance_mailer/slow_query_alert.html.erb -->
<!DOCTYPE html>
<html>
<head>
<style>
body { font-family: monospace; line-height: 1.6; }
.alert-header { background: #dc3545; color: white; padding: 15px; }
.alert-body { padding: 15px; }
.sql-query { background: #f8f9fa; padding: 10px; overflow-x: auto; }
.stack-trace { background: #fff3cd; padding: 10px; font-size: 12px; }
</style>
</head>
<body>
<div class="alert-header">
<h2>⚠️ Slow Query Detected</h2>
<p><strong><%= @environment.upcase %></strong> environment at <%= @timestamp.strftime('%Y-%m-%d %H:%M:%S') %></p>
</div>
<div class="alert-body">
<h3>Query Performance</h3>
<p><strong>Execution Time:</strong> <%= @duration %>ms (threshold: <%= @threshold %>ms)</p>
<h3>SQL Query</h3>
<div class="sql-query">
<code><%= @sql %></code>
</div>
<h3>Stack Trace</h3>
<div class="stack-trace">
<% @stack_trace.each do |line| %>
<div><%= line %></div>
<% end %>
</div>
<h3>Recommended Actions</h3>
<ul>
<li>Check if appropriate database indexes exist</li>
<li>Consider query optimization or N+1 prevention</li>
<li>Review recent code changes affecting this query</li>
<li>Run: <code>rake performance:check_indexes</code></li>
</ul>
</div>
</body>
</html>
These templates provide complete context for investigating performance problems without requiring log file access.
Database Performance Analysis Tools
Rake tasks automate performance analysis:
# lib/tasks/performance.rake
namespace :performance do
desc 'Check for missing database indexes'
task check_indexes: :environment do
puts 'Analyzing models for potential missing indexes...'
missing_indexes = []
ApplicationRecord.descendants.each do |model|
next if model.abstract_class?
# Check for foreign keys without indexes
model.reflections.each do |name, reflection|
if reflection.belongs_to?
foreign_key = reflection.foreign_key
unless has_index?(model, foreign_key)
missing_indexes << {
model: model.name,
column: foreign_key,
reason: "Foreign key (#{name})"
}
end
end
end
# Check for commonly queried columns
%w[created_at updated_at status type].each do |column|
if model.column_names.include?(column)
unless has_index?(model, column)
missing_indexes << {
model: model.name,
column: column,
reason: "Commonly queried column"
}
end
end
end
end
if missing_indexes.any?
puts "\nFound #{missing_indexes.count} potentially missing indexes:"
puts
missing_indexes.group_by { |idx| idx[:model] }.each do |model, indexes|
puts "#{model}:"
indexes.each do |idx|
puts " - #{idx[:column]} (#{idx[:reason]})"
puts " add_index :#{model.tableize}, :#{idx[:column]}"
end
puts
end
else
puts 'No obviously missing indexes found.'
end
end
desc 'Analyze slow queries from logs'
task analyze_queries: :environment do
log_file = Rails.root.join('log', "#{Rails.env}.log")
unless File.exist?(log_file)
puts "Log file not found: #{log_file}"
exit 1
end
slow_queries = []
File.readlines(log_file).each do |line|
if line.include?('[SLOW QUERY]')
match = line.match(/\[SLOW QUERY\] (\d+)ms/)
duration = match[1].to_i if match
# Extract SQL from next line
sql = line.split('SQL: ').last&.strip
slow_queries << { duration: duration, sql: sql } if duration && sql
end
end
if slow_queries.any?
puts "Found #{slow_queries.count} slow queries in logs:"
puts
# Group and sort by frequency
slow_queries.group_by { |q| q[:sql] }.sort_by { |_sql, queries| -queries.count }.each do |sql, queries|
avg_duration = queries.sum { |q| q[:duration] } / queries.count
puts "Occurrences: #{queries.count}, Avg: #{avg_duration}ms"
puts sql
puts
end
else
puts 'No slow queries found in logs.'
end
end
private
def has_index?(model, column)
model.connection.indexes(model.table_name).any? do |index|
index.columns.include?(column.to_s) ||
(index.columns.length == 1 && index.columns.first == column.to_s)
end
end
end
These tasks analyze application structure and runtime logs identifying optimization opportunities systematically.
Profiling Tools Integration
Advanced profiling requires additional gems:
# Gemfile
group :development do
gem 'flamegraph'
gem 'stackprof'
gem 'rack-mini-profiler'
end
# config/initializers/profiling.rb
if Rails.env.development?
require 'rack-mini-profiler'
# Rack::MiniProfiler shows profiling UI in development
Rack::MiniProfilerRails.initialize!(Rails.application)
end
Flamegraph generation for specific requests:
# To profile a specific request in development:
# Add ?pp=flamegraph to any URL to generate flamegraph
# Example: http://localhost:3000/properties?pp=flamegraph
# Generates flamegraph SVG showing time spent in each method
These tools provide deep CPU profiling when investigating specific performance bottlenecks requiring method-level analysis.
What's Next
The monitoring foundation enables sophisticated performance management: integrating with third-party APM services (Skylight, Scout, New Relic) for detailed analytics, implementing custom performance dashboards visualizing trends over time, adding database query plan analysis detecting inefficient execution plans, and implementing automatic performance regression detection in CI pipelines.
Future enhancements might include memory profiling detecting memory leaks, background job performance tracking monitoring Sidekiq job duration, API endpoint latency tracking with percentile calculations, and custom business metric instrumentation measuring feature-specific performance.
By implementing comprehensive performance monitoring infrastructure without expensive third-party dependencies, LetAdmin gains crucial visibility into production performance, enabling rapid identification and resolution of performance problems whilst maintaining complete control over monitoring data and infrastructure costs.