Apache RocketMQ Distributed Platform

Apache RocketMQ provides a foundation for distributed message queuing and data flow processing, boasting superior performance metrics, fault tolerance, capacity reaching trillions of messages, and highly elastic architectural scaling.

Key capabilities include:

Support for various interaction paradigms: pub/sub, request/response, and continuous stream processing.
Implementation of financially robust transactional messaging mechanisms.
Integrated high-availability features and intrinsic fault tolerance mechanisms, often leveraging DLedger.
Native message observability through built-in tracing, with compatibility for OpenTracing standards.
Broad integration points with established big-data pipelines and streaming environments.
Data retrieval capabilities based on defined timestamps or message offsets (retroactivity).
Guarantee of sequential message delivery (FIFO) and strict ordering guarantees within a single message queue.
Optimized mechanisms for both client-side retrieval (pull) and server-side delivery (push) of messages.
Ability to sustain millions of messages concurrently stored within an individual queue.
Native support for multiple communication protocols, including gRPC, MQTT, JMS, and OpenMessaging specifications.
A highly distributed architecture designed for seamless horizontal expansion.
Extremely rapid exchange capabilities for batched message transfers.
Diverse methods for message filtering, such as SQL expressions and tagging.
Pre-packaged Docker artifacts for simplified isolated deployment and cluster testing.
A comprehensive administrative interface offering configuration management, performance metrics visualization, and operational monitoring.
Security features including identity verification (authentication) and access control (authorization).
Provision of complimentary, open-source connectivity modules for data ingress and egress.
Support for lightweight, near real-time computation tasks.

Initial Setup Guide

This section details the procedures for installing RocketMQ across different operational environments. For iterative local testing, a singular instance of each core component suffices.

Local Deployment

RocketMQ functions across all primary operating systems, contingent only upon the presence of Java JDK version 8 or newer. Verification is done via the command: shell $ java -version java version "1.8.0_121"

Windows operators should procure the 4.9.3 binary archive here: Download Link. After extraction (e.g., to D:\rocketmq), proceed.

For Linux/macOS users, utilize the following shell commands: shell

Obtain the package from an Apache mirror

$ wget https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip

Decompress the archive

$ unzip rocketmq-all-4.9.3-bin-release.zip

Open a command-line interface, navigate into the extracted directory's bin folder: shell $ cd rocketmq-4.9.3/bin

1) Activate the NameServer Component

The NameServer will initialize listening on port 9876 on all interfaces (0.0.0.0). Ensure this port is uncontested locally, then execute:

For macOS and Linux environments: shell

Initiate Name Server operation

$ nohup sh mqnamesrv &

Verify successful startup using logs

$ tail -f ~/logs/rocketmqlogs/namesrv.log The Name Server boot success...

For Windows users, environment variable configuration is mandatory: - Right-click the 'Computer' icon on the desktop and select 'Properties'. - Navigate to 'Advanced system settings'. - Access 'Environment Variables'. - Introduce a new system variable: Key ROCKETMQ_HOME, Value D:\rocketmq.

After setting the variable, return to the RocketMQ root directory in the command prompt and run: shell $ mqnamesrv.cmd The Name Server boot success...

2) Activate the Broker Component

For macOS and Linux users: shell

Initiate Broker operation

$ nohup sh bin/mqbroker -n localhost:9876 &

Confirm Broker readiness (e.g., IP 192.168.1.2, Name broker-a)

$ tail -f ~/logs/rocketmqlogs/broker.log The broker[broker-a, 192.169.1.2:10911] boot success...

For Windows users: shell $ mqbroker.cmd -n localhost:9876 The broker[broker-a, 192.169.1.2:10911] boot success...

Containerized Execution with Docker

RocketMQ can be instantiated within Docker containers. The --net=host configuration is utilized to expose the required internal ports directly to the host machine.

1) NameServer Launch

shell $ docker run -it --net=host apache/rocketmq ./mqnamesrv

2) Broker Launch

Persistent storage mapping is defined for the message store: shell $ docker run -it --net=host --mount source=/tmp/store,target=/home/rocketmq/store apache/rocketmq ./mqbroker -n localhost:9876

Orchestrated Deployment via Kubernetes

Deployment of a RocketMQ cluster within a Kubernetes environment is facilitated by the RocketMQ Operator. Prerequisite: kubectl utility configured with appropriate kubeconfig credentials.

1) Installation of Custom Resource Definitions (CRDs) shell

Clone the operator repository

$ git clone https://github.com/apache/rocketmq-operator $ cd rocketmq-operator && make deploy

Verify CRD registration

$ kubectl get crd | grep rocketmq.apache.org brokers.rocketmq.apache.org 2022-05-12T09:23:18Z consoles.rocketmq.apache.org 2022-05-12T09:23:19Z nameservices.rocketmq.apache.org 2022-05-12T09:23:18Z topictransfers.rocketmq.apache.org 2022-05-12T09:23:19Z

Confirm operator pod health

$ kubectl get pods | grep rocketmq-operator rocketmq-operator-6f65c77c49-8hwmj 1/1 Running 0 93s

2) Cluster Resource Provisioning shell

Instantiate the RocketMQ cluster definition

$ cd example && kubectl create -f rocketmq_v1alpha1_rocketmq_cluster.yaml

Inspect cluster state (StatefulSets)

$ kubectl get sts NAME READY AGE broker-0-master 1/1 107m broker-0-replica-1 1/1 107m name-service 1/1 107m

Community Engagement and Auxiliary Projects

RocketMQ Streams: A lightweight stream processing engine built on top of Apache RocketMQ.
RocketMQ Flink: Connector for Apache Flink, supporting both source and sink operations within stream and tabular data contexts.
Multi-Language Client Libraries:
RocketMQ Spring: Facilitates rapid integration of Apache RocketMQ with Spring Boot applications.
RocketMQ Exporter: A utility to export RocketMQ metrics for consumption by Prometheus.
RocketMQ Operator: Provides Kubernetes deployment automation for RocketMQ clusters.
RocketMQ Docker: Source repository for the official Apache RocketMQ container images.
RocketMQ Dashboard: The console interface for system administration, monitoring, and configuration.
RocketMQ Connect: A scalable and dependable utility for streaming data between RocketMQ and external systems.
RocketMQ MQTT: A modernized MQTT architecture implementation, enhancing RocketMQ's capability to interface with IoT endpoints and mobile applications.
RocketMQ Incubating Projects: External projects in incubation, including components like logappender, configuration via rocketmq-ansible, monitoring with rocketmq-beats-integration, and standards adherence via rocketmq-cloudevents-binding, among others.
RocketMQ Site: Repository hosting the content for the official Apache RocketMQ website.

Resources and Communication Channels

Mailing Lists: https://rocketmq.apache.org/about/contact/
Primary Website: https://rocketmq.apache.org
Documentation (Quick Start): https://rocketmq.apache.org/docs/quick-start/
Issue Tracking: https://github.com/apache/rocketmq/issues
Improvement Proposals (RIPs): https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal
Community Q&A: https://stackoverflow.com/questions/tagged/rocketmq
Instant Messaging: https://rocketmq-invite-automation.herokuapp.com/ (Slack invitation link)

Participation Guidelines

We actively encourage external contributions, ranging from minor textual refinements to the implementation of significant new features (refer to RIPs for major proposals) or other recognized forms of support. Detailed contribution instructions are available here.

Licensing

Governed under the terms of the Apache License, Version 2.0. Copyright is held by the Apache Software Foundation.

Export Compliance Advisory

This software package incorporates cryptographic modules. Individuals must verify local jurisdictional regulations concerning the acquisition, utilization, and potential cross-border transfer of encryption technology prior to deployment. Consult http://www.wassenaar.org/ for supplementary information.

The U.S. Department of Commerce, Bureau of Industry and Security (BIS), has classified this software under Export Commodity Control Number (ECCN) 5D002.C.1, applicable to information security software employing asymmetric cryptographic functions. Distribution by the Apache Software Foundation qualifies for export under the License Exception ENC Technology Software Unrestricted (TSU), detailed in EAR Section 740.13, covering both compiled and source code forms.

Specific details on the integrated cryptography:

This platform utilizes Apache Commons Crypto (https://commons.apache.org/proper/commons-crypto/) to facilitate service-to-service authentication, data encryption, and decryption across network boundaries.

(Note: The subsequent text regarding XMLHttpRequest is deemed irrelevant context from the original README and is omitted for conciseness while maintaining the technical focus of the entry.)

rocketmq-messaging-system

Author

Gchenxx

Quick Info

Actions

Tags