Multiple failures in production

Hi all,

We’ve installed Consul from scratch, version 1.0.0. We find that some features that work perfectly on development, make Consul crash on production. We have both environments installed in the same machine (for testing purposes), same version of PostgreSQL, Rubi (and its gems), etc. Some of the failures we found are:

  • When trying to validate a new user, production crashes leaving those errors in the production log:

W, [2019-10-02T13:54:42.020252 #12696] WARN – : [8c6935e3-8ea0-4739-948d-c9e7e83e7389] [Rollbar] Got unexpected status code from Rollbar api: 400
F, [2019-10-02T13:54:42.022170 #12696] FATAL – : [8c6935e3-8ea0-4739-948d-c9e7e83e7389]
F, [2019-10-02T13:54:42.022336 #12696] FATAL – : [8c6935e3-8ea0-4739-948d-c9e7e83e7389] Errno::ENOENT (No such file or directory @ rb_sysopen - ):
F, [2019-10-02T13:54:42.022432 #12696] FATAL – : [8c6935e3-8ea0-4739-948d-c9e7e83e7389]
F, [2019-10-02T13:54:42.022529 #12696] FATAL – : [8c6935e3-8ea0-4739-948d-c9e7e83e7389] lib/census_api.rb:61:in get_response_body' [8c6935e3-8ea0-4739-948d-c9e7e83e7389] lib/census_api.rb:7:inblock in call’
[8c6935e3-8ea0-4739-948d-c9e7e83e7389] lib/census_api.rb:6:in each' [8c6935e3-8ea0-4739-948d-c9e7e83e7389] lib/census_api.rb:6:incall’
[8c6935e3-8ea0-4739-948d-c9e7e83e7389] lib/census_caller.rb:7:in call' [8c6935e3-8ea0-4739-948d-c9e7e83e7389] app/models/verification/residence.rb:74:inretrieve_census_data’
[8c6935e3-8ea0-4739-948d-c9e7e83e7389] app/models/verification/residence.rb:28:in save' [8c6935e3-8ea0-4739-948d-c9e7e83e7389] app/controllers/verification/residence_controller.rb:13:increate’

This works perfectly on development.

  • We can’t even sign out in production. Consul also crashes, leaving the following in the production log:

W, [2019-10-02T18:08:22.937421 #12693] WARN – : [c1c6fd44-1e76-4507-9613-24c9d6d0e551] [Rollbar] Got unexpected status code from Rollbar api: 400
F, [2019-10-02T18:08:22.941364 #12693] FATAL – : [c1c6fd44-1e76-4507-9613-24c9d6d0e551]
F, [2019-10-02T18:08:22.941475 #12693] FATAL – : [c1c6fd44-1e76-4507-9613-24c9d6d0e551] ActiveRecord::RecordNotFound (Couldn’t find User with ‘id’=sign_out [WHERE “users”.“hidden_at” IS NULL
]):
F, [2019-10-02T18:08:22.941551 #12693] FATAL – : [c1c6fd44-1e76-4507-9613-24c9d6d0e551]
F, [2019-10-02T18:08:22.941673 #12693] FATAL – : [c1c6fd44-1e76-4507-9613-24c9d6d0e551] activerecord (5.0.7.2) lib/active_record/relation/finder_methods.rb:353:in raise_record_not_found_exc eption!' [c1c6fd44-1e76-4507-9613-24c9d6d0e551] activerecord (5.0.7.2) lib/active_record/relation/finder_methods.rb:479:infind_one’
[c1c6fd44-1e76-4507-9613-24c9d6d0e551] activerecord (5.0.7.2) lib/active_record/relation/finder_methods.rb:458:in find_with_ids' [c1c6fd44-1e76-4507-9613-24c9d6d0e551] activerecord (5.0.7.2) lib/active_record/relation/finder_methods.rb:66:infind’
[c1c6fd44-1e76-4507-9613-24c9d6d0e551] activerecord (5.0.7.2) lib/active_record/querying.rb:3:in find' [c1c6fd44-1e76-4507-9613-24c9d6d0e551] activerecord (5.0.7.2) lib/active_record/core.rb:151:infind’
[c1c6fd44-1e76-4507-9613-24c9d6d0e551] cancancan (2.3.0) lib/cancan/model_adapters/abstract_adapter.rb:20:in find' [c1c6fd44-1e76-4507-9613-24c9d6d0e551] cancancan (2.3.0) lib/cancan/controller_resource_finder.rb:11:infind_resource’
[c1c6fd44-1e76-4507-9613-24c9d6d0e551] cancancan (2.3.0) lib/cancan/controller_resource_loader.rb:106:in load_resource_instance' [c1c6fd44-1e76-4507-9613-24c9d6d0e551] cancancan (2.3.0) lib/cancan/controller_resource_loader.rb:15:inload_resource’
[c1c6fd44-1e76-4507-9613-24c9d6d0e551] cancancan (2.3.0) lib/cancan/controller_resource.rb:31:in `load_and_authorize_resource’

We can provide full logs if requested. We activated debug output in PostgreSQL, and the last query is:
SELECT “site_customization_pages”.* FROM “site_customization_pages” WHERE “site_customization_pages”.“status” = ‘published’ AND “site_customization_pages”.“slug” = ‘sign_out’ ORDER BY id DESC LIMIT 1;

So we can’t even understand what has to do the table “users” in this error. By the way, it works perfectly in development too.

Any help will be greatly appreciated. We can provide all information that could be of any help. Our main question is: why does it work in development, but crash in production? We deploy using Capistrano.

Hi TomelloSoulMan,

Have you solved the problems you ask in this Post?
If so, congratulations!
If you still have problems please contact me and we will review those errors.

Greetings

Hi @rockandror, we opened an issue in GitHub and it’s now working ok. If you want to review it, it’s in https://github.com/consul/consul/issues/3742