cucumber/cucumber-ruby

Summary formatter should print more error information

Closed

#1,513 opened on Mar 25, 2021

 (9 comments) (0 reactions) (0 assignees)Ruby (1,142 forks)batch import
:pray: help wanted:white_check_mark: accepted:zap: enhancementgood first issue

Repository metrics

Stars
 (5,159 stars)
PR merge metrics
 (Avg merge 11d 2h) (7 merged PRs in 30d)

Description

Describe the bug

Hi! Not sure if this is intentional but I was expecting the summary formatter to format "success" the way it does, but in the case of errors, I was expecting to still give me useful information about the errors.

To Reproduce

  • Clone https://github.com/activeadmin/activeadmin.

  • Run bundle install.

  • Introduce a typo bug like

    diff --git a/spec/support/templates/policies/post_policy.rb b/spec/support/templates/policies/post_policy.rb
    index 8d39cac13..8480071cc 100644
    --- a/spec/support/templates/policies/post_policy.rb
    +++ b/spec/support/templates/policies/post_policy.rb
    @@ -1,5 +1,5 @@
     class PostPolicy < ApplicationPolicy
       def update?
    -    record.author == user
    +    recorda.author == user
       end
     end
    
  • Setup test environment with bin/rake setup.

  • Run cucumber feature with bin/cucumber --format summary features/authorization_pundit.feature:21.

Expected behavior

I was expecting a more helpful error like the one you get with --format pretty:

$ bin/cucumber --format pretty features/authorization_pundit.feature:21 
Using the default profile...
@authorization
Feature: Authorizing Access using Pundit

  Background:                            # features/authorization_pundit.feature:4
    Given I am logged in                 # features/step_definitions/user_steps.rb:9
    And 1 post exists                    # features/step_definitions/factory_steps.rb:6
    And a configuration of:              # features/step_definitions/configuration_steps.rb:13
      """
      require 'pundit'

      ActiveAdmin.application.namespace(:admin).authorization_adapter = ActiveAdmin::PunditAdapter

      ActiveAdmin.register Post do
      end

      ActiveAdmin.register_page "No Access" do
      end
      """
    And I am on the index page for posts # features/step_definitions/web_steps.rb:52
      undefined local variable or method `recorda' for #<PostPolicy:0x0000558c61339148 @user=#<AdminUser id: 1, email: "admin@example.com", created_at: "2021-03-25 17:23:41.114977000 +0000", updated_at: "2021-03-25 17:23:41.114977000 +0000">, @record=#<Post id: 1, title: "Hello World 0", body: nil, published_date: nil, author_id: nil, position: nil, custom_category_id: nil, starred: nil, foo_id: nil, created_at: "2021-03-25 17:23:41.141817000 +0000", updated_at: "2021-03-25 17:23:41.141817000 +0000">>
      Did you mean?  record
                     @record (ActionView::Template::Error)
      ./tmp/test_apps/rails_61/app/policies/post_policy.rb:3:in `update?'
      ./lib/active_admin/pundit_adapter.rb:17:in `public_send'
      ./lib/active_admin/pundit_adapter.rb:17:in `authorized?'
      ./lib/active_admin/base_controller/authorization.rb:38:in `authorized?'
      ./lib/active_admin/base_controller/authorization.rb:20:in `authorized?'
      ./lib/active_admin/views/index_as_table.rb:379:in `defaults'
      ./lib/active_admin/views/index_as_table.rb:362:in `block (2 levels) in actions'
      ./lib/active_admin/views/index_as_table.rb:361:in `block in actions'
      ./lib/active_admin/view_helpers/display_helper.rb:58:in `find_value'
      ./lib/active_admin/view_helpers/display_helper.rb:45:in `format_attribute'
      ./lib/active_admin/views/components/table_for.rb:102:in `block in build_table_cell'
      ./lib/active_admin/views/components/table_for.rb:101:in `build_table_cell'
      ./lib/active_admin/views/components/table_for.rb:45:in `block (2 levels) in column'
      ./lib/active_admin/views/components/table_for.rb:44:in `block in column'
      ./lib/active_admin/views/components/table_for.rb:43:in `each_with_index'
      ./lib/active_admin/views/components/table_for.rb:43:in `column'
      ./lib/active_admin/views/index_as_table.rb:354:in `actions'
      ./lib/active_admin/views/index_as_table.rb:264:in `block in default_table'
      ./lib/active_admin/views/index_as_table.rb:249:in `instance_exec'
      ./lib/active_admin/views/index_as_table.rb:249:in `block in build'
      ./lib/active_admin/views/components/table_for.rb:21:in `build'
      ./lib/active_admin/views/index_as_table.rb:254:in `table_for'
      ./lib/active_admin/views/index_as_table.rb:247:in `build'
      ./lib/active_admin/views/pages/index.rb:141:in `block (2 levels) in render_index'
      ./lib/active_admin/views/pages/index.rb:140:in `block in render_index'
      ./lib/active_admin/views/pages/index.rb:133:in `render_index'
      ./lib/active_admin/views/pages/index.rb:50:in `build_collection'
      ./lib/active_admin/views/pages/index.rb:28:in `block in main_content'
      ./lib/active_admin/views/pages/index.rb:36:in `wrap_with_batch_action_form'
      ./lib/active_admin/views/pages/index.rb:26:in `main_content'
      ./lib/active_admin/views/pages/base.rb:101:in `block (2 levels) in build_main_content_wrapper'
      ./lib/active_admin/views/pages/base.rb:100:in `block in build_main_content_wrapper'
      ./lib/active_admin/views/pages/base.rb:99:in `build_main_content_wrapper'
      ./lib/active_admin/views/pages/base.rb:83:in `block in build_page_content'
      ./lib/active_admin/views/pages/base.rb:82:in `build_page_content'
      ./lib/active_admin/views/pages/base.rb:59:in `block (2 levels) in build_page'
      ./lib/active_admin/views/pages/base.rb:55:in `block in build_page'
      ./lib/active_admin/views/pages/base.rb:54:in `build_page'
      ./lib/active_admin/views/pages/base.rb:9:in `build'
      ./app/views/active_admin/resource/index.html.arb:2:in `block in __home_deivid__ode_activeadmin_app_views_active_admin_resource_index_html_arb___1907912821212394310_26820'
      ./app/views/active_admin/resource/index.html.arb:1:in `new'
      ./app/views/active_admin/resource/index.html.arb:1:in `__home_deivid__ode_activeadmin_app_views_active_admin_resource_index_html_arb___1907912821212394310_26820'
      ./lib/active_admin/resource_controller/streaming.rb:12:in `index'
      ./features/step_definitions/web_steps.rb:53:in `/^I am on (.+)$/'
      features/authorization_pundit.feature:19:in `I am on the index page for posts'

  Scenario: Attempt to access a resource I am not authorized to see   # features/authorization_pundit.feature:21
    When I go to the last post's edit page                            # features/step_definitions/web_steps.rb:56
    Then I should see "You are not authorized to perform this action" # features/step_definitions/web_steps.rb:80

Failing Scenarios:
cucumber features/authorization_pundit.feature:21 # Scenario: Attempt to access a resource I am not authorized to see

1 scenario (1 failed)
6 steps (1 failed, 2 skipped, 3 passed)
0m0.607s

Randomized with seed 9856

However, with --format summary, all the error information is "swallowed", so issues are hard to debug:

$ bin/cucumber --format summary features/authorization_pundit.feature:21 
Authorizing Access using Pundit
  Attempt to access a resource I am not authorized to see ✗

Failing Scenarios:
cucumber features/authorization_pundit.feature:21 # Scenario: Attempt to access a resource I am not authorized to see

1 scenario (1 failed)
6 steps (1 failed, 2 skipped, 3 passed)
0m0.726s

Randomized with seed 18864

Context & Motivation

I introduced some bugs in my code that made cucumber scenarios failed, but I had no clue on how to go about investigating them until I realized changing the formatter would give me more useful information for debugging.

Your Environment

  • Versions used [5.3.0]
  • Operating System and version [Linux]

Contributor guide