How To Restrict File Search In Sublime Based On Project

This is a documentation on how to restrict text search to within specific directories per project you are working on in Sublime.

You may often find it annoying that a simple text search is searching in folders that is not part of your source code. While you can easily flip the switch in the user settings of the sublime text editor, it might not be ideal if you wear multiple hats like me and work on different frameworks.

One framework’s trash may be another’s treasure. Folders that are considered junk in one framework might be important in another. And if we happen to work on these framework together simultaneously, we would have to constantly flip the switch on and off as we jump between working on these projects.

sublime meme | vic-l

If you are using sublime text because the framework you are working on is simple enough to manage and you do not want the computation-heavy indexing and compilation process to be running in the background constantly, this is an article for you to boost your productivity.

The User Setting Way

The commonly documented way of configuring your sublime editor is to tweak the configuration option in the user setting. Using restricted file search as an example, we can simple press CMD + , (assuming you are on a mac) to call up the user setting files in the sublime editor.

Next add this setting under the Preferences.sublime-settings file as shown:

{
  ...
  "binary_file_patterns": [
    "node_modules/*",
    "public/packs/",
    "public/assets/",
    "public/packs-test/",
    "tmp/*",
    "*.jpg",
    "*.jpeg",
    "*.png",
    "*.gif",
    "*.ttf",
    "*.tga",
    "*.dds",
    "*.ico",
    "*.eot",
    "*.pdf",
    "*.swf",
    "*.jar",
    "*.zip"
  ]
  ...
}

The binary_file_patterns option will instruct Sublime to treat these files as binary files. Binary files are not readable by human, hence it is not considered in Goto Anything or Find in Files functions by default.

Line 4 to 8 are folders that we want excluded from our search process.

The rest are referring to specific files base on their extensions.

This setup is what I typically use for my Rails projects.

The Project-specific Way

The better alternative is to base the setting on the project level so that we do no overlap the settings between projects.

Save the project as a sublime project via File > Save As…

If moving the mouse is not your thing, can you simply create a file in the root directory with the .sublime-project extension.

Next, add the same binary_file_patterns setting as shown previously under the folders key as shown:

{
  "folders":
  [
    {
      "path": ".",
      "binary_file_patterns": [
        ...
      ]
    }
  ],
}

Line 5, the path key is required. This is added by default when we save our project as a sublime project. More information on its function and purpose can be found here.

Next, close sublime. This time, open our project via the sublime project file that you have created.

Make a search now and you will realise that you are no longer searching in the folders that you have no interest in. The search process is also much faster than before because the program is going through less files, and ignoring the bulkier ones.

Hallelujah!

Housekeeping

Once we created the .sublime-project file, another file with the extension .sublime-workspace will also be created. The latter contains user specific data and you will not want to share it with other developers who may be working on the same source code as you. Add this file to our <code>.gitignore</code> file to achieve this.

Setup Bootstrap In Rails 6 With Webpacker For Development And Production

This is a documentation on how to setup Bootstrap 4 in Rails 6 using Webpacker. As the framework shifts away from sprockets and the asset pipeline to embrace the dominating methodology of handling frontend affairs in the Javascript world that is webpack, we have to adapt along.
The way to setup a css framework to bootstrap your application has undergone a revamp, and this article seeks to cover the essential steps to set it up.

Pre-requisites

This article will assume you have set up all the required tools required for a typical Rails 6 application.

The main extra tool you will need as compared to previous versions of Rails is the yarn package manager. You can install yarn on your computer via various ways based on your preference and your OS.

We will not be covering it in this article.

Setting Up Bootstrap

With the shift in paradigm of handling front end assets, we no longer install front end libraries using gems. In the past, these gems are merely wrappers around the Javascript libraries and files which present a number of problems.

First, the latest changes in the Javascript world will take some time to propagate into the Rails realm.

Second, having an intermediate wrapper increase the potential points of failure during the wrapping process.

Third, we are really dependent on the angels who are working on these wrappers. If they do not update the gems frequently, we are stuck with the old features. This can be frustrating if you are waiting for a certain bug fix or a new feature that is already available in the latest release.

To install bootstrap, run this command.

yarn add bootstrap jquery popper.js

This command will automatically install the latest bootstrap package in the yarn registries and add its dependency entry and version in your package.json file. Jquery and popper.js are libraries that bootstrap depends on, especially in their Javascript department.

The JS And CSS Files

The main Javascript file, application.js should now reside in the app/javascript/packs folder. This is because Webpacker will now look for all the javascript files in this directory to compile. This is the default setting for Webpacker.

Of course, you can go ahead and change the configuration to your liking. However, keep in mind that Rails promotes convention over configuration. This implies that as much as possible, methodologies and practices should follow a certain default unless absolutely necessary. this has multiple advantages. My favorite one is the portability of code among fellow Rails developers. Developers can easily understand the flow of logic and where to find bugs because they are where are expected to be. This cuts down the development time and cost greatly.

The application.js file should look like this:

require("@rails/ujs").start()
require("turbolinks").start()
require("@rails/activestorage").start()
require("channels")
require("bootstrap")

// stylesheets
require("../stylesheets/main.scss")

Line 1 to 4 are the default files already present in the file.

Line 5 adds the Bootstrap Javascript library.

Line 8 adds your custom stylesheet. Now, this file can be placed anywhere. In the above example, the path is relative to where the application.js file is. Hence, the file is placed in app/javascript/stylesheets/main.scss in this example.

Next, we import the Bootstrap stylesheet files in the main stylesheet file.

@import "bootstrap/scss/bootstrap";

Note that we are importing files from the node_modules folder, and not a bootstrap folder placed in the relative path of the current directory of the main stylesheet file.

Also, you do not need the ~ in front of the path to signify that it is from the node_modules folder like you would usually do for other non-Rails project using webpack. The tilde alias in webpack is a default webpack configuration that will resolve to the node_modules folder. While it will still work here, it is not required as the node_modules folder is already configured as part of the search paths that webpack will look for when resolving the modules.

Now, you may be wondering how to the Bootstrap libraries will work without importing any of its dependencies, that are popper.js and Jquery. We will come to that in a minute. Before that, let’s look at the views.

The Views

Now, we will need to add the javascript and stylesheets files into the page. Following convention in this example, we will add to the application.html.erb layout so that the Bootstrap framework can be accessed in all pages. These lines of code are added in the head section of the layout template.

<%= stylesheet_pack_tag 'application' %>
<%= javascript_pack_tag 'application', 'data-turbolinks-track': 'reload' %>

There are a  number of things that are different from the old implementation.

Line 1 adds the compiled stylesheets path that webpacker will compile. Note that this only happens if the extract_css option is set to true in the webpacker.yml file. More about this later.

As you can see, there is no more stylesheet_include_tag. In the past, this helper method will get files from the public/assets folder, into which the asset pipeline will compile stylesheets and javascript files with other added pre and post processing. Now, everything is going to be done by Webpack.

Here what’s happening.

Webpack will look at application.js and find the stylesheet files that are included in it. Then, using a combination of Webpack loaders, Webpack will know how to compile and translate the scss syntax, the url paths of assets used etc. into a css file that the browser can read and implement its styling.

These Webpack loaders are already included by the Webpacker and its configurations set up. However, there are many loaders out there that are not included by default. They tend to be less used conventionally and will require manual intervention from your side.

One example is using ruby code inside your javascript files. This requires the rails-erb-loader that will “teach” Webpack to understand the erb syntax. The implementation involves a number of steps, one of which is to append this loader to the Webpack environment.js configuration file. Thankfully, for this case, the community has deemed it a pretty common use case that there is, at least, a rake task that comes together with the Webpacker gem to set this up easily.

The compilation process mentioned above, however, is not applied in the development environment by default. This is due to the extract_css settings in the webpacker.yml page. More about this and its implications in a bit.

Note that stylesheet_include_tag still works for assets you place in the app/assets folder. However, while that is true, as Rails moves away from the old Sprockets and assets pipeline convention, this is expected to become deprecated in the future.

The Webpacker Configuration File

Lastly, we need to add the dependencies of bootstrap. This takes place in the config/webpack/environment.js file.

const { environment } = require('@rails/webpacker')
const webpack = require('webpack')

environment.plugins.append('Provide', new webpack.ProvidePlugin({
  $: 'jquery',
  jQuery: 'jquery',
  Popper: ['popper.js', 'default']
}))

module.exports = environment

As you can see, we are utilising the ProvidePlugin function of Webpack to add the dependency libraries in all the javascript packs instead of having to import them everywhere.

This is just an example of how we can import files with Webpack in Rails. And in this case, especially for jQuery, it makes a lot of sense as there is a high chance that we will be using it in other javascript files.

The extract_css Option

There is one last point I would like to touch on. That is the extract_css option in the config/webpacker.yml file.

When set to true, webpack will compile the stylesheet files that were imported into the javascript files into external standalone stylesheets. These compiled files will then be added into the views via the stylesheet_pack_tag helper method as mentioned earlier.

In comparison, when set to false, the stylesheets are not compiled into standalone files. Instead, they are added into the view as a blob during runtime by the the relevant javascript file. This takes place only after the javascript file has been completely downloaded by the browser.

In development mode, the conventional setting for the extract_css option is false, and this has quite a significant implication on how the website will behave.

One, there might be a flash of unstyled content (FOUC) when the page loads because the javascript files are loaded asynchronously. This is unlike the css files which are blocking resources that will pause the rendering of the website until the file has been downloaded. This asynchronous loading of files allows the website to continue rendering while it waits for itself to be completely downloaded before computing the css blob and insert it into the html source code. If the web page loads before this occurs, the style for the web page is not present, and FOUC will thus occur.

Two, the stylesheet_pack_tag is not needed in the development environment using the default setting. Things will seem to work fine only until it is pushed into the production environment where the extract_css option is set to true, desirably and by default. So make sure to use the stylesheet_pack_tag helper even if it seems to work fine without it in development.

Conclusion

At this point of time, the application should be running with Bootstrap in place. Do test out how it will differ in the production environment as compared to development.

JWT With Refresh Token Using Devise And Doorkeeper Without Authorization

This is a documentation on setting up the authentication system of a rails project in a primarily API environment.

Rails is essentially a framework for bootstrapping applications on the web environment. The support for APIs is thus lacking. One aspect of it is an off the shelf authentication system that can fit both the API and web environment on the same monolith application.

The Devise gem, while hugely popular and has established itself as the de facto authentication gem in the Rails world, does not come supported with an authentication system fit for interaction via APIs. The main reason is because it relies on cookies, which is strictly a browser feature.

To overcome this, often, we have to use other gems to couple with it to leverage on its scaffolded features for user authentications.

In this article, we will use the Doorkeeper and Devise combination to provide an authentication using JSON Web Tokens (JWT), the modern day best practices for authentication via APIs.

But let us first understand what kind of authentication system we are building and why we choose Doorkeeper.

The Example Authentication System​

Now, as a disclaimer, there are many ways to setup an authentication system.

One such consideration is the devise-jwt gem, which serves as a direct replacement to the cookies for your APIs. It is simple to implement and allows you to choose from multiple strategies to expire your token. Except that it does not come with a refresh token.

This implies that the token will expire and the user will have to login again. If your application requires such security, you can consider this gem instead.

However, in this article, the authentication system that I will like to set up is one that allows user to log in via JWT that will expire, and upon expiry, the front end can use the refresh token to get a new JWT without having the user to login again. This allows the user to stay logged in without compromising security excessively.

Why do we need to ensure the JWT expires?

Security Considerations Using JWT

Allowing user to be logged in permanently is kind of the standard user flow for many applications nowadays. The easiest way to implement this is to not expire the JWT. However, that is a recipe for disaster. It is akin to passing your password around when making API requests. And the moment it gets compromised, malicious attackers can have all the time in the world to explore your account and even plan their attacks, and leaving the users all the time in the world to say their prayers.

We thus have to enforce expiry on the JWT at the very least. To accomplish that without forcing the user to have to login again is to use a refresh token.

A refresh token stays in the local machine for the whole of it lifetime, or until the user actively logs out. This allows that the access token, which is dispatched out into the wild wild west otherwise known as the Internet, can at least expire within a certain period of time. And when it expires, the front end can use the refresh token to get a new access token to allow the user to continue its current session as though he or she is still logged in. So even if the access token gets compromised in the world beyond the walls, the potential damage is reduced.

This mechanism is made into a standard known as Oauth. There are many libraries out there that implements this already, and it is widely adopted among many of the software products that we use like Google account, facebook and twitter.

However, while this works with authenticating with these external providers, it has a crucial requirement that we do not want when implementing our own in house authentication system (I am referring to the old school email and password login). That step is the authorization step.

Some of us may have come across such a  request when we try to sign up with an app via Facebook, as shown below:

oauth authorization | vic-l

While this feature is absolutely essential in the OAuth protocol. it presents an awkwardness when we want to leverage on the OAuth libraries to implement JWT with refresh token for our in house authentication.

The Awkwardness Of OAuth

Just make sure we are on the same page, here are a summary of the points that led up to this awkwardness.

First, we need to make the tokens expire for security reasons.

Second, refresh token are here to the rescue, and they are used in the OAuth protocol.

Third, unfortunately, OAuth requires an authorization step, which in house authentication system do not need.

Last, we cannot leverage on the various OAuth implementation out there to implement a JWT with refresh token without having to hack these libraries and somehow sidestep the authorization step.

Hacking Doorkeeper

The OAuth library that we will be using is Doorkeeper. Its wiki page already has a section on skipping the authorization step, which certainly signals the demand for such an implementation. However, there are some points missing from this implementation and this article will try to cover more of them. These steps are highly influenced by this blog post.

First, install doorkeeper and its migration files, following its instructions.

rails g doorkeeper:install
rails g doorkeeper:migration

Changes To The Migration Files

Edit the migration file like this.

# frozen_string_literal: true

class CreateDoorkeeperTables < ActiveRecord::Migration[6.0]
  def change
    create_table :oauth_access_tokens do |t|
      t.references :resource_owner, index: true
      t.integer :application_id
      t.text :token, null: false
      t.string :refresh_token
      t.integer :expires_in
      t.datetime :revoked_at
      t.datetime :created_at, null: false
      t.string :scopes
    end

    # required to allow model.destroy to work
    create_table :oauth_access_grants do |t|
      t.references :resource_owner, null: false
      t.integer :application_id
      t.string   :token, null: false
      t.integer  :expires_in, null: false
      t.text     :redirect_uri, null: false
      t.datetime :created_at, null: false
      t.datetime :revoked_at
      t.string   :scopes, null: false, default: ''
    end

    # Uncomment below to ensure a valid reference to the resource owner's table
    add_foreign_key :oauth_access_tokens, :users, column: :resource_owner_id
  end
end

Compared to the original generated copy of the migration file, we have removed the oauth_applications table which refers to the application that we want to grant permission to in the authorization step. Since we are skipping the authoirzation, there is no need to have this unused table.

Next we have changed

t.references :application, null: false

into

t.integer :application_id

Since the table is no longer present, we cannot use the references helper, and need to resort to specifying the the basic data type. We are still keeping this column in the database although we have deleted the application table because Doorkeeper uses this attribute while running its operation. Without it, an error will occur along the lines of “column not found“.

In fact, we also do not need the oauth_access_grants table, which is the bridge between the oauth_access_tokens table and the oauth_applications. It records which token authorized which application. However, without it, an error will be thrown when destroying a user record from the database. If you do not have such a feature, feel free to remove this table as well.

Lastly, only keep the foreign key implementation on oauth_access_tokens and change the model name according to whatever you have named your model.

Changes To The Initializer File

Edit the configuration in the doorkeeper initializer file as such:

# frozen_string_literal: true

Doorkeeper.configure do
  ...
  resource_owner_from_credentials do |routes|
    user = User.find_for_database_authentication(email: params[:email])
    request.env['warden'].set_user(user, scope: :user, store: false)
    user
  end
  ...
  use_refresh_token
  ...
  grant_flows %w[password]
  ...
  skip_authorization do
    true
  end
  ...
  api_only
  base_controller 'ActionController::API'
end

We are essentially following this documentation on their wiki, but with some additional content and some slight changes, to implement an authentication flow whereby the token is returned in exchange for the credentials of the resource owner, in this case the user’s email and password.

Line 5 to 9 is the main implementation.

On line 6, we are instructing Doorkeeper to use Devise method, find_for_database_authentication, for authenticating the correct user. This method will run use the underlying warden gem in Devise to do its authentication magic. This, however, will save the user in the session, which can be a problem when we check for sessions in the controller level. More on this later. We undo this in line 7.

On line 7, we instruct warden to set the user only for the request and not store it in the session, as documented here.

On line 11, uncomment use_refresh_token to ensure a refresh token is generated on login.

Line 13 is for older version of Doorkeeper at 2.1+. More information in the above mentioned wiki page.

Line 15 to 17, we instruct Doorkeeper to skip the authorization step.

Line 19, we set mode to api_only. This can help to optimize the application to a certain extent. For example, it skips forgery protection checks that is not necessary in an API environment, which reduces computational requirement and latency.

Line 20, I am just explicitly setting the base controller to use ActionController::API instead of the default ActionController::Base, although this should have already been implemented when the mode is set to api_only.

Controller Level

Devise comes with a helper method, current_user or whatever your model name is, to access the current authenticated resource. This, however, will return a nil value in the current implementation because the underlying method will not be working. The underlying method is, taken from the source code:

def current_#{mapping}
  @current_#{mapping} ||= warden.authenticate(scope: :#{mapping})
end

With reference to this stackoverflow answer, we will modify it to look like this:

def current_user
  @current_user ||= if doorkeeper_token
                      User.find(doorkeeper_token.resource_owner_id)
                    else
                      warden.authenticate(scope: :user, store: false)
                    end
end

We have essentially overwritten the default implementation by Devise to check for the “current_user” using the doorkeeper_token first, and fallback on the default implementation. The fallback will be useful in the event where our application will still be using the traditional login methods via a web browser. Feel free to remove it if you are not going to have such any request coming from a web browser. And of course, remember to handle the scenario of a nil doorkeeper_token.

Last but not least, implement that authorization check at the correct routes and actions in the Doorkeeper::TokensController via the before_action callback like how you would when using just Devise alone.

before_action :doorkeeper_authorize!

Custom Controller

I personally have some custom code that I want to add to all my APIs so that when the frontend consumes my APIs, they will not be left stunned by responses having different JSON structure.

I keep a response_code and a response_message in all my APIs for the frontend to react accordingly and trigger the desired UX flow.

Here is how I modify my controller. Let’s start off with some modification to the Doorkeeper modules.

module Doorkeeper
  module OAuth
    class TokenResponse
      def body
        {
          # copied
          "access_token" => token.plaintext_token,
          "token_type" => token.token_type,
          "expires_in" => token.expires_in_seconds,
          "refresh_token" => token.plaintext_refresh_token,
          "scope" => token.scopes_string,
          "created_at" => token.created_at.to_i,
          # custom
          response_code: 'custom.success.default',
          response_message: I18n.t('custom.success.default')
        }.reject { |_, value| value.blank? }
      end
    end
  end
end

Here, I modify the response from Doorkeeper to add in my required keys. I am using I18n to handle the custom messages and prepare the application for a global audience.

Next, the error response. By default, Doorkeeper returns the keys error and error_description. That is different from what I want. I will overwrite it totally.

module Doorkeeper
  module OAuth
    class ErrorResponse
      # overwrite, do not use default error and error_description key
      def body
        {
          response_code: "doorkeeper.errors.messages.#{name}",
          response_message: description,
          state: state
        }
      end
    end
  end
end


name, description and state are accessible variables in the default class. I integrate them into my custom API response for standardization purpose.

Now the controller. There are 3 main methods: login, refresh and logout. Let’s go through them.

module Api
  module V1
    class TokensController < Doorkeeper::TokensController
      before_action :doorkeeper_authorize!, only: [:logout]

      def login
        user = User.find_for_database_authentication(email: params[:email])

        case
        when user.nil? || !user.valid_password?(params[:password])
          response_code = 'devise.failure.invalid'
          render json: {
            response_code: response_code,
            response_message: I18n.t(response_code)
          }, status: 400
        when user&.inactive_message == :unconfirmed
          response_code = 'devise.failure.unconfirmed'
          render json: {
            response_code: response_code,
            response_message: I18n.t(response_code)
          }, status: 400
        when !user.active_for_authentication?
          create
        else
          create
        end
      end

      def refresh
        create
      end

      def logout
        # Follow doorkeeper-5.1.0 revoke method, different from the latest code on the repo on 6 Sept 2019

        params[:token] = access_token

        revoke_token if authorized?
        response_code = 'custom.success.default'
        render json: {
          response_code: response_code,
          response_message: I18n.t(response_code)
        }, status: 200
      end

      private

      def access_token
        pattern = /^Bearer /
        header = request.headers['Authorization']
        header.gsub(pattern, '') if header && header.match(pattern)
      end
    end
  end
end

Firstly, I am applying the doorkeeper_authorize! callback on the logout method only as that is the only method that will require the user to be logged in.

The login method will largely follow what we defined in the initializer file under the resource_owner_from_credentials block. The modification here is to define specific error scenarios and their respective response_code here. For those scenarios that are of no interest to me, I will leave it to the catch-all case and and return what is now the default modified ErrorResponse.

The second case in particular is specific to my project. I allow admin users to create the users, and have a flag (created_by_admin_and_authenticated) to differentiate them.

  • nil means the user registered normally
  • false means they are created by the admin user, but have yet to authenticate with the email that our server sent out to them
  • true means they are created by admin user and have also authenticated their email address

I will force users who are created by admin users but have yet to authenticate via email to reset their password, leveraging on what Devise has already provided with its password module.

Note: this is definitely much to be optimized here. For example, the find_for_database_authentication method is being called twice here for a successful user login, once in this custom controller and the other in the default Doorkeeper::TokensController create method.

The refresh method to refresh the access_token is practically the same as the default create method, but I am overriding it here because I use ApiPie to add documentation to the routes. For those who do not use ApiPie, we define its required parameters, headers etc. above the line 31 to define the documentation for the refresh method. I also can rename the route in doing so to create an API that the front end developers that I am working with would find more familiar with.

The logout method makes use of the revoke_token method, according to its source code, to revoke the JWT.

In my application, I require my frontend to add the JWT token in the Authorization header instead of a parameter in the request body based on convention. Doorkeeper, on the other hand, expects the token to be present in the params. To overcome this, I created the custom private access_token method to get the token in the header that the front end has placed in their requests. That token is then placed in the params object behind the key named token as Doorkeeper would have expected. Doorkeeper can then do its thing without having to modify any of its internal workings.

Since the revoke_token method provided by Doorkeeper will make use of the token key in the params, I will first use the private access_token method to extract the JWT token from the Authorization header. Then add it as the value to the token key of the params variable.

The logout method is required for the front end to dispose of the current access token they have for security purposes. I also use it to remove the users’ devices token so that they do not receive push notifications after logging out.

Login Request

{
	"email": "user1@test.com",
	"password": "user1@test.com",
	"grant_type": "password"
}


A login request will have these keys. In particular, the grant_type strategy used should be password.

Conclusion

You should be able to login with the correct credentials with the default Doorkeeper::TokensController and access your controllers with the correct resource, just like how you would when using Devise alone. Otherwise, you can use your custom controller inherit and customise the authentication routes, as I have demonstrated.

Hope this was helpful!